Data Forecasting and Trends
Forecasting is the concept of knowing the future using the past. The technique is as old as humans, as our ancestors have predicted and marked many events in the calendar such as summer solstice and winter solstice using past astronomical data. The applications of forecasting are vast and many fields including business, finance, retail etc., use it to predict the future using the historical data. Forecasting is the old cousin of intelligent forecasting [UC1] which is used in many big organizations to predict the future trends in sales, marketing research, profit margins, new product development etc.,
As I have been writing about too many futuristic topics involving digital transformation trends of big data, IoT, AI, AR/VR, Data analytics,5G etc., I thought that it would be appropriate for me to take a step back to later-half of the nineties, (before Cloud and Big Data) and wanted to bring you the insights of how businesses were handling forecasting using simple probabilistic methods and staying ahead of the competition. Hence the topic.
The major benefits of data forecasting include:
1) Draw inference from the past and present data
2) Measure performance using KPI metrics
3) To predict the future trends, which might help in planning for the future.
There are three distinct levels of forecasting: Organizational Level, Industrial level, National level
There are also three types of forecasting, when accuracy is specified, as data projection [UC2] varies with time: 1) Short Term 2) Medium Term 3) Long Term
The top ten methods of muted data forecasting included sampling, surveys, and biases, estimating parameters, casual models, micro econometric data analysis, A/B testing and experimentation and evaluation, qualitative model, probabilistic simulation model and time series projection model, though the intelligent forecasting includes predictive analytics, time series forecasting and analysis, linear multiple and ANOVA regression models, regression diagnostics.
Among the ten, the top three mostly used data forecasting methods by many data forecasting firms are qualitative, time series projection and casual models as many of the other techniques can be fit into these three major headings:
· Qualitative projection models
Use a series of dedicated events, which generated a favorable outcome is fed as input data. The input might include a load of past events too, but it is necessary to do it.
· Time series projection models
Uses historical data from past and find favorable pattern and pattern changes are used to draw a forecasting model. The top three models are
1. Smoothing techniques are of many varieties and give versatility in projecting complex tasks. But the disadvantage of some varieties of smoothing is the large computational time taken to calculate the projection points.
2. Box Jenkins method is the most expensive of the time series projection models as it is widely used in financial applications. Because of the high accurate predictions and nature of mathematical model used, they are the most time-consuming methods too.
3. X11 technique is useful for predicting short term events using analytical methods and data spanning over three years.
There is one other model which uses time series projection, and it is used for trend forecasting than data-based forecasting. The main difference between data forecasting and data trend forecasting is the procedure involved. The first step involves drawing a trend line based on the time series data from the past and the present and then this trend line is matched to a mathematical equation. The mathematical equation can be calculus based (slope/ area characteristics) or logarithm [UC3] based (exponent based functions characterized by ovals, Hyberbolas etc., ). This trend-line forecasting used for any new product forecasts ranges for a medium term of 3 months to 2 years and for a long-term projection of 2 years or more. The disadvantage with trend line forecasting is the data requirement and it needs at least 5 years of historical data to start with as it deals with long term forecasting.
· Casual projection models
This uses specific factors which impacts the forecasting system [UC4] as well as the forecasted element such as performance, profits, value, returns etc., This is similar to both qualitative and time series projections models, as it considers, dedicated events and historical past data into account, while considering the input.
1. The most basic of the casual models is the regression model which casually relates two variables using regression technique of finding relationship between two given entities and plotting the data. Similar to trend line forecasting, the plotted data can be expressed in terms of an equation involving an independent variable x and a dependent variable y, such as y = 2.3 x +1.2, this gives a direct relationship between two variables y and x and hence the forecast can be made, based on one variable x using this equation. This is a good technique for short term forecasting but performs poorly in long term forecasting. Similar to trend line technique, this needs a good amount of historical data and hence quite expensive.
2. A better form of regression model is econometric model, uses the same technique as the regression model, but the mathematical equations are precise as the number of points plotted is too high and hence results in an accurate forecasting in many fields of sales, new product research, market research etc., The input –output analysis model on the other hand, deals with the flow of goods and services into a department and out of the department. The main problem with this analysis model is the expense involved in procuring accurate historical data and based on the accuracy levels obtained, the expenditure will vary from $100,000 to $300,000 US dollars.
3. Diffusion Index is another method also used in product sales which is the percentage of the economic indicators. The up and down influx of the indicators are noted down over the period of time and averaged and multiplied by one hundred to get this diffusion index. There are other important methods in casual models too as described in the image.
The selection of a method depends on a variety of factors –
1. Field of Application- Financial performance evaluation, marketing and research forecast, sales and profit evaluation for the fiscal year, warehouse inventory performance evaluation needed for purchase etc.,
2. Context and the relevance- Context refers to the life cycle (Research, design, implementation, deployment etc.,) of the product for which the forecast made or which quarterly performance, the forecast is made for etc., and the relevance is how the organization is related to the forecast whether as a customer or a vendor or a stakeholder or the core business itself
3. Purpose of the forecast -Customer requirements for the ask, where it is going to be of use, such as to convince the investors, convince the customers, convince the executives etc.,
4. Availability of data -Data is the most important part of the forecast and thus it is vital to use the available data than use the costly option of creating data from the available data to fill in the unavailable information to improve accuracy.
5. History of forecast — Is there any historical precedence in the organization for the type of forecast made? How have the previous forecasts been useful to the organization? How productive has the organization become because of the use of the previous forecasts?
6. The degree of accuracy required -The precision of the forecast is linked to the outcome of the forecast.
7. Overall value of the forecast — Value is calculated in business [monetary] and cultural [face value of the organization] terms
8. The cost /benefit ratio of the forecast- This gives an idea to the organization to decide whether to go ahead with the forecasting and help in decision making for other related forecasts too, do not forget, the cost of accuracy increases the cost of forecast.
9. The time needed and available to make the forecast — It is not advisable to quote a wrong delivery time for a forecasting project, as time is the most essential characteristics of any forecast, a forecast made for the second term, if gets completed in the third term is of no use. Therefore, use the right amount resources (human, capital and operational) to state the right time needed to make a forecast. If the time available does not meet the time needed, then it is not worth going ahead with the forecast, as it is a waste.
As per a Harvard business review from archive, the choice of the forecasting technique is decided by the forecaster based on the best use of the available data. Two major pitfalls the forecasters must avoid are
1) Not to use another technique using predicted data based on the available data, because it provides increasingly accurate forecasts.
2) Not to use an enhanced method to improve the accuracy of the forecasting using non-existent data obtained using expensive cost.
Consequently, it is important to understand the product life cycle stages for which the forecast is made. For instance, if the product has five stage life cycle (Research and Development, Design and Engineering and Deployment, Growth, Maturity, and Decline) then the forecasting must consider the correct stages for which the forecast is made to avoid any errors done during the previous stage forecast impacting the successive stages, when used as input data. The life cycle analysis varies depending on the life cycle, because product development life cycle mentioned above and product market life cycle, (innovators, early adopters, early majority, laggards etc.,) are completely different.
Before I conclude this article, I want to briefly mention some of the techniques used in financial forecasting as against business forecasting. Besides, business forecasting being used for financial forecasting, the precision model needs increased refinement, especially when forecasting revenue, profits, and turnovers to investors hence they are done separately.
Thus, any forecasting technique standard or intelligent used to predict future possibilities involves a good amount of accurate historical data and better needs of the business. As I mentioned above, the data in the past was gathered through surveys, questionnaires and past performance, and in the recent times, the same is being gathered through sensors placed suitably inside the devices, thus extending this forecasting methods from business indictors to industrial and engineering applications such as predictive maintenance, optimization of systems etc., Many applications uses the digital twin in order to arrive at the required result as digital revolution is happening around the world, currently, stretching the possibilities of the forecasting to entirely newer levels.
[UC1] A type of predictive analytics, where the data patterns and models are created using supervised and unsupervised machine learning algorithms.
[UC2] I can explain this concept with one simple example: Weather forecasting- short term is hourly forecast, medium term is 24 hours forecast and long term is 7-day forecast. Typically, the term period varies depending on the applications. For business forecasts, the short term is 3 months, medium term is 1–2 years and long term 5 years and beyond.
[UC3] Logarithm in mathematics is the inverse function of exponentials. Exponentials are represented as a base. The natural logarithms are to the base of e and represented as ln. Also, there is a logarithm to the base of 10 represented as log. The general representation is log base number = base number.
[UC4] The software and hardware used to create the forecast
