Introducing a Data Blending API (Support) in Cube.js

Introducing Data Blending API inCube.js

Sometimes in our daily data visualization, we need to merge
several similar data sources so that we can manipulate everything
as one solid bunch of data.

For example, we may have an omnichannel shop where online and
offline sales are stored in two tables. Or, we may have similar
data sources that have only a single common dimension: time. How
can we calculate summary metrics for a period? Joining by time is
the wrong way because we can’t apply granularity to get the
summary data correctly.

Furthermore, how can we find seasonal patterns from summarized
metrics? And how can we get and process data synchronously to track
correlations between channels?

Well, the new data blending functionality in version 0.20.0 of
Cube.js takes care of all these cases.

If you’re not familiar with Cube.js yet, please have
a look at this
. It will show you how to set up the database, start a
Cube.js server, and get information about data schemes and
analytical cubes.

Please, keep in mind that we used here another dataset:

$ curl > ecom2-dump.sql$ createdb ecom$ psql --dbname ecom -f ecom2-dump.sql

Now let’s dive into the metrics for an example shop and
visualize sales by channel and as a summary.

is the full source and live demo of
the example.

I used React to implement this example, but querying in Cube.js
works the same way as in Angular, Vue, and vanilla JS.

Our schema has two cubes:


cube(`Orders`, { sql: `SELECT * FROM public.orders`, measures: {   count: {     type: `count`,   }, }, dimensions: {   id: {     sql: `id`,     type: `number`,     primaryKey: true,   },   createdAt: {     sql: `created_at`,     type: `time`,   }, },});

and OrdersOffline.js

cube(`OrdersOffline`, { sql: `SELECT * FROM public.orders_offline`, measures: {   count: {     type: `count`,   }, }, dimensions: {   id: {     sql: `id`,     type: `number`,     primaryKey: true,   },   createdAt: {     sql: `created_at`,     type: `time`,   }, },});

The existence of at least one-time dimension in each cube is a
core requirement for merging the data properly. In other words, the
data are suitable for blending only if you can present the data on
a timeline. Sales statistics or two lists of users that both have
an account created date are appropriate datasets for data blending.
However, two lists of countries with only a population value
can’t be united this way.

A Special Query Format for Data Blending

A simple and minimalistic approach is to apply data blending to
a query object when we retrieve data from our frontend application.
The schema and backend don’t need to be changed.

const { resultSet } = useCubeQuery([   {     measures: ['Orders.count'],     timeDimensions: [       {         dimension: 'Orders.createdAt',         dateRange: ['2022-01-01', '2022-12-31'],         granularity: 'month',       },     ],   },   {     measures: ['OrdersOffline.count'],     timeDimensions: [       {         dimension: 'OrdersOffline.createdAt',         dateRange: ['2022-01-01', '2022-12-31'],         granularity: 'month',       },     ],   }, ]);

The blended data is an array of query objects, so we just
combine regular Cube.js query objects into an array with a defined
dateRange and

As a result, Cube.js returns an array of regular resultSet

But what if we want to do calculations over blended data sources
or create custom metrics? For example, how can we define ratios
calculated using data from two sources? How can we apply formulas
that depend on data from multiple sources?

In this case, we can use another data blending function. We
start by setting up a new cube.

Data Blending Implementation within a Schema

Let’s create AllSales.js inside the schema folder:

cube(`AllSales`, { sql: ` select id, created_at, 'OrdersOffline' row_type from ${OrdersOffline.sql()} UNION ALL select id, created_at, 'Orders' row_type from ${Orders.sql()} `, measures: {   count: {     sql: `id`,     type: `count`,   },   onlineRevenue: {     type: `count`,     filters: [{ sql: `${CUBE}.row_type = 'Orders'` }],   },   offlineRevenue: {     type: `count`,     filters: [{ sql: `${CUBE}.row_type = 'OrdersOffline'` }],   },   onlineRevenuePercentage: {     sql: `(${onlineRevenue} / NULLIF(${onlineRevenue} + ${offlineRevenue} + 0.0, 0))*100`,     type: `number`,   },   offlineRevenuePercentage: {     sql: `(${offlineRevenue} / NULLIF(${onlineRevenue} + ${offlineRevenue} + 0.0, 0))*100`,     type: `number`,   },   commonPercentage: {     sql: `${onlineRevenuePercentage} + ${offlineRevenuePercentage}`,     type: `number`,   }, }, dimensions: {   createdAt: {     sql: `created_at`,     type: `time`,   },   revenueType: {     sql: `row_type`,     type: `string`,   }, },});

Here we’ve applied a UNION statement to blend data from two
tables, but it’s possible to combine even more.

Using this approach, we can easily define and combine values
from several blended data sources. We can even use
calculated values
and SQL formulas.

We can retrieve data from frontend applications and process the
results in the usual way:

const { resultSet: result } = useCubeQuery({   measures: [     'AllSales.onlineRevenuePercentage',     'AllSales.offlineRevenuePercentage',     'AllSales.commonPercentage',   ],   timeDimensions: [     {       dimension: 'AllSales.createdAt',       dateRange: ['2022-01-01', '2022-12-31'],       granularity: 'month',     },   ], });


If we need to visualize data from several sources and apply time
granularity to the data, then with data blending, we need to write
less code and we can simplify the application logic.

We looked at two ways to implement data blending: We retrieved
data as an array of query objects from a frontend application. This
is simple to do and the schema doesn’t need to be modified. We
can even merge data from several databases. Furthermore, we can
retrieve and process independent data synchronously so that we can
visualize it on a timeline. We blended data by defining a special
cube in a schema. This approach allows us to apply aggregate
functions to all sources simultaneously and we can define
calculated values.

We hope this tutorial will help you to write less code and help
to build more creative visualizations. If you have any questions or
feedback or you want to share your projects, please use our
Slack channel or mention us
at Twitter.

Also, don’t forget to sign up for our monthly newsletter to
get more information about Cube.js updates and releases.

Originally published at
on October 1, 2020.

Introducing a Data Blending API (Support) in Cube.js
originally published in Cube
on Medium, where people are continuing the conversation by
highlighting and responding to this story.

Time Series Forecasting: KNN vs. ARIMA

It is always hard to find a proper model to forecast time series
data. One of the reasons is that models that use time-series data
often expose to serial correlation. In this article, we will
compare k nearest neighbor (KNN) regression which is a supervised
machine learning method, with a more classical and stochastic
process, autoregressive integrated moving average (ARIMA).

We will use the monthly prices of refined gold
futures(XAUTRY) for one gram in Turkish Lira
traded on BIST(Istanbul Stock Exchange) for
forecasting. We created the data frame starting from 2013. You can
download the relevant excel file from

#building the time series data

df_xautry <- read_excel("xau_try.xlsx")
xautry_ts <- ts(df_xautry$price,start = c(2013,1),frequency
= 12)

KNN Regression

We are going to use tsfknn package which can be
used to forecast time series in R programming
. KNN regression process consists of
instance, features, and targets components. Below
is an example to understand the components and the process.

pred <- knn_forecasting(xautry_ts, h = 6, lags =
autoplot(pred, highlight = "neighbors",faceting = TRUE)

The lags parameter indicates the lagged values
of the time series data. The lagged values are used as features or
explanatory variables. In this example, because our time series
data is monthly, we set the parameters to 1:12. The last 12
observations of the data build the instance, which
is shown by purple points on the graph.

This instance is used as a reference vector to
find features that are the closest vectors to that
instance. The relevant distance metric is calculated by the
Euclidean formula as shown below:

\sqrt {\displaystyle\sum_{x=1}^{n} (f_{x}^i-q_x)^2 }

q_x denotes the instance and f_{x}^i indicates the features that are ranked in order by
the distance metric. The k parameter determines the number of k
closest features vectors which are called k nearest

nearest_neighbors function shows the instance,
k nearest neighbors, and the targets.

#Lag 12 Lag 11 Lag 10  Lag 9  Lag 8  Lag 7  Lag 6  Lag 5  Lag
4  Lag 3  Lag 2
#272.79 277.55 272.91 291.12 306.76 322.53 345.28 382.02
384.06 389.36 448.28
# Lag 1
#  Lag 12 Lag 11 Lag 10  Lag 9  Lag 8  Lag 7  Lag 6  Lag 5 
Lag 4  Lag 3  Lag 2
#1 240.87 245.78 248.24 260.94 258.68 288.16 272.79 277.55
272.91 291.12 306.76
#2 225.74 240.87 245.78 248.24 260.94 258.68 288.16 272.79
277.55 272.91 291.12
#3 223.97 225.74 240.87 245.78 248.24 260.94 258.68 288.16
272.79 277.55 272.91
#   Lag 1     H1     H2     H3     H4     H5     H6
#1 322.53 345.28 382.02 384.06 389.36 448.28 462.59
#2 306.76 322.53 345.28 382.02 384.06 389.36 448.28
#3 291.12 306.76 322.53 345.28 382.02 384.06 389.36

Targets are the time-series data that come
right after the nearest neighbors and their number is the value of
the h parameter. The targets of the nearest
neighbors are averaged to forecast the future h periods.

\displaystyle\sum_{i=1}^k \frac {t^i} {k}

As you can see from the above plotting, features or targets
might overlap the instance. This is because the time series data
has no seasonality and is in a specific uptrend. This process we
mentioned so far is called
MIMO(multiple-input-multiple-output) strategy that
is a forecasting method used as a default with KNN.

Decomposing and analyzing the time
series data

Before we mention the model, we first analyze the time series
data on whether there is
The decomposition analysis
is used to calculate the strength of
seasonality which is described as shown below:

#Seasonality and trend measurements
fit <- stl(xautry_ts,s.window = "periodic",t.window =
13,robust = TRUE)
seasonality <- fit %>% seasonal()
trend <- fit %>% trendcycle()
remain <- fit %>% remainder()
#[1] 0.990609
#[1] 0.2624522

The stl function is a decomposing time series
method. STL is short for seasonal and trend decomposition using
loess, which loess is a method for estimating nonlinear
relationships. The t.window(trend window) is the
number of consecutive observations to be used for estimating the
trend and should be odd numbers. The s.window(seasonal window) is
the number of consecutive years to estimate each value in the
seasonal component, and in this example, is set to
'periodic' to be the same for all years. The
robust parameter is set to 'TRUE'
which means that the outliers won't affect the estimations of trend
and seasonal components.

When we examine the results from the above code chunk, it is
seen that there is a strong uptrend with 0.99, weak seasonality
strength with 0.26, because that any value less than 0.4 is
accepted as a negligible seasonal effect. Because of that, we will
prefer the non-seasonal ARIMA model.

Non-seasonal ARIMA

This model consists of differencing with autoregression and
moving average. Let's explain each part of the model.

Differencing: First of all, we have to explain
stationary data. If data doesn't contain
information pattern like trend or seasonality in other words is
white noise that data is stationary. White noise
time series has no autocorrelation at all.

Differencing is a simple arithmetic operation that extracts the
difference between two consecutive observations to make that data


The above equation shows the first differences that difference
at lag 1. Sometimes, the first difference is not enough to obtain
stationary data, hence, we might have to do differencing of the
time series data one more time(second-order

In autoregressive models, our target variable
is a linear combination of its own lagged variables. This means the
explanatory variables of the target variable are past values of
that target variable. The AR(p) notation denotes
the autoregressive model of order p and the
\boldsymbol\epsilon_t denotes the white noise.

y_t=c + \phi_1 y_{t-1}+ \phi_2y_{t-2}+...+\phi_py_{t-p}+\epsilon_t

Moving average models, unlike autoregressive
models, they use past error(white noise) values for predictor
variables. The MA(q) notation denotes the
autoregressive model of order q.

y_t=c + \theta_1 \epsilon_{t-1}+ \theta_2\epsilon_{t-2}+...+\theta_q\epsilon_{t-q}

If we integrate differencing with autoregression and the moving
average model, we obtain a non-seasonal ARIMA model which is short
for the autoregressive integrated moving average.

y_t' is the differenced data and we must remember it may
have been first and second order. The explanatory variables are
both lagged values of y_t and past forecast errors. This is denoted as
ARIMA(p,d,q) where p; the order
of the autoregressive; d, degree of first
differencing; q, the order of the moving

Modeling with non-seasonal ARIMA

Before we model the data, first we split the data as train and
test to calculate accuracy for the ARIMA model.

#Splitting time series into training and test data
test <- window(xautry_ts, start=c(2019,3))
train <- window(xautry_ts, end=c(2019,2))

#ARIMA modeling
fit_arima<- auto.arima(train, seasonal=FALSE,
#Series: train
#ARIMA(0,1,2) with drift
#          ma1      ma2   drift
#      -0.1539  -0.2407  1.8378
#s.e.   0.1129   0.1063  0.6554
#sigma^2 estimated as 86.5:  log likelihood=-264.93
#AIC=537.85   AICc=538.44   BIC=547.01

As seen above code chunk, stepwise=FALSE, approximation=FALSE
parameters are used to amplify the searching for all possible model
options. The drift component indicates the
constant c which is the average change in the
historical data. From the results above, we can see that there is
no autoregressive part of the model, but a second-order moving
average with the first differencing.

Modeling with KNN

#Modeling and forecasting
pred <- knn_forecasting(xautry_ts, h = 18, lags =

#Forecasting plotting for KNN
autoplot(pred, highlight = "neighbors", faceting = TRUE)

Forecasting and accuracy comparison
between the models

#ARIMA accuracy
f_arima<- fit_arima %>% forecast(h =18) %>%
#                 RMSE       MAE      MAPE     
#Training set  9.045488  5.529203  4.283023
#Test set     94.788638 74.322505 20.878096

For forecasting accuracy, we take the results of the test set
shown above.

#Forecasting plot for ARIMA
fit_arima %>% forecast(h=18) %>% autoplot()+

#KNN Accuracy
ro <- rolling_origin(pred, h = 18,rolling = FALSE)
#  RMSE       MAE      MAPE
#137.12465 129.77352  40.22795

The rolling_origin function is used to evaluate
the accuracy based on rolling origin. The rolling
parameter should be set to FALSE which makes the
last 18 observations as the test set and the remaining as the
training set; just like we did for ARIMA modeling before. The test
set would not be a constant vector if we had set the rolling
parameter to its default value of TRUE. Below, there is an example
for h=6 that rolling_origin parameter set to TRUE. You can see the
test set dynamically changed from 6 to 1 and they eventually build
as a matrix, not a constant vector.

#Accuracy plot for KNN

When we compare the results of the accuracy measurements like
RMSE or MAPE, we can easily see that the ARIMA model is much better
than the KNN model for our non-seasonal time series data.

The original article is


Post-scripting to Deal with Complex SQL Queries

SQL (or the stored procedure) can handle most of the database
computations. If the computations are complex or hard to deal in
SQL, we use another programming language to read data out of the
database to manipulate it. Such a programming language handles the
data read and manipulation with a simple script. So we call the
process the post-SQL scripting.

The scenarios that SQL is not good at handling include complex
set-based operations, order-based operations, associative
operations and multi-step computations, etc. Due to SQL’s
incomplete set orientation and lack of explicit set data type,
it’s almost impossible to reuse the intermediate sets generated
during the computation. The forced aggregate after each grouping
operation makes it impossible to use the post-grouping subsets. The
unordered-set-based SQL adopts a very roundabout way to handle
order-based computations, like inter-row (group) computations and
ranking operations. The language generates temporary sequence
numbers using JOIN(s) or subqueries, making the program hard to
write and slow to compute. Record reference is another SQL’s
incapability. The language uses a subquery or a JOIN statement to
express the relationship. Code becomes ridiculously complicated
when there are multiple data levels or when self-joins are needed.
SQL doesn’t foster multi-step coding. Programmers have to write
very long query containing layers of subqueries. Though stored
procedures can alleviate the problem, they are not always
available. DBA has strict rules about the privileges of using
stored procedures, and old and small databases don’t support
stored procedures. Besides, it’s inconvenient to debug a stored
procedure. This makes it unsuitable to do a procedural

There are other scenarios that require post-SQL scripting. To
migrate the algorithm between different database products or
between database and non-relational database, data source or output
target isn’t the database but a file, and mixed computation
performed between multiple databases, for example. All these
external database computations need post-SQL scripting.

The most important role of a post-SQL script is to achieve the
complex computations that SQL is not good at. It would be better if
they had some advanced features, such as the ability of processing
data of various sources (files and non-relational databases, for
example) and handling a relatively large amount of data, and
satisfactory performance. But, the basic thing is that the
scripting tool should be able to perform database read/write

The popular post-SQL scripting tools are Java, Python pandas and
esProc. Now let’s look at and examine their scripting


High-level languages, such C++ and Java, are theoretically
almighty and thus are able to manage computations SQL is hard to
deal with. Java supports generic type and has relatively
comprehensive set orientation to handle complex set-based
computations. A Java array has intrinsic sequence numbers to
implement order-based computations conveniently. Java can express a
relationship using object reference and handle join operations
well. It also supports procedural syntax, including branch and
loop, to achieve complex multi-step computations.

Unfortunately, Java lacks class libraries for structured
computations. It hardcodes even the simplest structured
computations and creates the most basic data type manually. That
makes Java code lengthy and complicated.

Here’s an example of order-based computation: get the number
of longest consecutively rising trading days for a stock. Database
AAPL stores a stock’s price information in fields including
transaction dates and closing price.

In an intuitive way, we loop through the records ordered by
dates, add 1 to the number of consecutive rising days (the initial
value is 0) if the closing price of the current record is greater
than that of the previous one; and compare the number of
consecutively rising days with the current largest number of
consecutively rising days (the initial value is 0) to get the new
largest number and reset the number as 0. Repeat the process until
the loop is over and the current largest number is the final
desired one.

SQL can’t implement the intuitive algorithm because it
doesn’t support order-based computations. But it has its own
tricks. It groups stock records ordered by dates. The way is like
this: put records whose closing prices rise consecutively in one
group, that is, group the current record with the previous one if
its closing price rises and create a new group if its price lowers;
then count the members in every group and find the largest number,
which is the largest number of consecutively rising days.

Examples of such SQL code can be found

Red Herren: SEC Commissioner shovels climate nonsense at
NYTimes readers

SEC Commissioner Allison Herren Lee has an op-ed in today’s New
York Times cheerleading for enhanced corporate disclosure of
climate risks. She has no idea what’s she’s talking about.
Below is my line-by-line take down. And don’t forget about my SEC
petition to stop climate lying. As the climate crisis intensifies,
[‘Intensified’? How is that? …

Personal updates and DSC

After more than 10 years being involved with Data Science
Central, initially as the founder, and most recently being acquired
by TechTarget, I have
decided to pursue new interests. TechTarget now has a great team
taking care of DSC, and I will still be involved as a consultant to
make sure that everything continues to run smoothly and that the
quality standards are maintained and even enhanced. In my new role,
I will become a contributor and write articles for DSC, so I will
still be visible, even more than before as I find more time to
write more articles.

I am happy to announce that TechTarget has just brought on Kurt
Cagle as the new Community Manager for DSC. Kurt is a former DSC
blogger and also living in WA two miles away from my place where
DSC was created (so it will be easy to meet in person to explain
all the tricks) and will take over many of my responsibilities,
including interactions with the community. As for me, I will be
spending more needed time in my new restaurant opening next month,
in Anacortes, WA.

My new upcoming articles will be original, with the same style
aimed at explaining sometimes advanced and new, original ML
concepts and recipes in layman terms to a large range of analytics
professionals: data scientists, executives, decision makers, and
consumers of analytic products.

I wish all the best to Kurt and TechTarget. I am very happy to
have worked with the stellar TechTarget team after the acquisition,
as well as to continue to work with former DSC colleagues who were
hired by TechTarget and are still there and happy today. This
relationship will continue to grow in the foreseeable future, with
TechTarget’s strong commitment to the DSC community.



Weekly Digest, September 28

Monday newsletter published by Data Science Central. Previous
editions can be foundhere.
The contribution flagged with a + is our selection for the picture
of the week. To subscribe, follow
this link


Featured Resources and Technical

Featured Articles

Picture of the Week

Source: article flagged with a + 

To make sure you keep getting these emails, please
add to your address book or
whitelist us. To subscribe, click here.
Follow us: Twitter | Facebook.

6 Best AI-Based Apps in 2020: A Brief Introduction

Artificial Intelligence (AI) is one of the best recognized and
most visible technologies in this time of science and technology.
It allows accomplishing the task in less time more reliably and
consistently. This innovative technology enables you to simplify
your activities, thinking skills, speed the process, and make the
traditional system more genius.

AI is a technology that, if used in the proper direction, will
benefit your business, making you more efficient in your activities
and essentially assist you in making money.

AI is being used to develop respectively mobile and web-based
applications. In the past years, AI-controlled solutions or
software for iOS and Android are increasingly emerging in the
market. Excessive use of smartphones is a crucial factor in the
development of robust AI applications.

In this article, we will give you brief details about some of
the successful AI-based apps available in the market.

Below Are 6 Best AI-Based Apps for iOS and

  • Cortana: Cortana is a virtual assistant developed by
    Microsoft. This application is currently the most successful
    AI-based app. By using Cortana, you can keep your calendar and
    schedule up to date, create and maintain lists, set a reminder and
    alarms, find facts, definitions and information, and many more.
    This application syncs your mobile and your computer. It can also
    respond to questions using data from the Bing search engine. For
    example, weather and traffic conditions, sports scores,
    biographies, etc.
  • Wysa: Without a question, with Wysa, you
    can initiate a discussion with strangers. It's an AI-driven chatbot
    and incredibly friendly when you're far from everyone. This
    application allows you to start a conversation with an unfamiliar
    person with confidence. Wysa also helps you to address a variety of
    issues, including depression, anxiety, sleep, problems regarding
    the LGBTQ+ community, and much more.
  • ELSA: ELSA is an AI-powered application
    which works only as a Digital English Tutor. Users can learn to
    speak English more fluently without much of a push using this
    application. ELSA utilizes patented speech technology with Machine
    Learning (ML) and Artificial Intelligence (AI) to identify people's
    pronunciation errors.
    • ELSA has a variety of highlights that are required to reach
      a moderate level of intelligence in English. This application has
      voice recognition technology. This stunning feature allows you to
      interact with the necessary American literary pronunciation in
  • FaceApp: FaceApp is a photo-morphing
    application developed by Facebook. This application is an
    incredibly well-known AI-driven Android and iOS platforms. It helps
    to transform the actual photos into an amazing one. With the help
    of this smart application, you can change the overall appearance,
    such as a smile, a haircut, and give a unique look.
  • Google Allo: With this app, user can send
    messages without typing. The user has to tell Google Allo, in
    particular, what to write in a message. By this, you can send
    unlimited messages free of charge to anyone. Google Allo also acts
    as a browser interface and runs on Google Chrome, Mozilla Firefox
    and Opera.
  • Meitu: Meitu is one of the most magnificent
    beauty filter applications that can make your selfies more
    gorgeous. This application modifies your photos using its built-in
    advanced AI algorithms. You can also include messages, blur
    backgrounds, and add filters. Young ladies are particularly
    attracted to this application as it has a variety of makeup options
    that allow them to hide acne, trim the face, firm the skin, widen
    the eyes, and remove dark circles.


In this cutting edge world, AI is turning into a
helpful tool for an enormous number of companies. A few major
companies have begun using AI-based software. Likewise, a gigantic
number of small firms have additionally begun to put resources in
AI to improve their competitiveness through effectiveness,
innovation, and profitability.

Remember, after finalizing the idea that you will be
working on, the next step is to connect with the best mobile app
development company which will design and develop a mobile app.
Also, you have to update yourself regarding the latest market trend
and analyze the future trend.

Common Technical Characteristics of Some Major Stock Market

In this blog, I will be covering some charts based on the
technical data of three major stock market corrections: the Crash
of 1987, Dot-com Correction of 2000, and the Great Crash of 1929.
By technical data, I mean that the charts are produced using stock
market prices.  More than a month ago, I wrote about opening my
first stock trading account in 1987; this is the same year I was
hired for my first regular full-time job.  I worked as a store
clerk on Bay Street, which is arguably the Canadian equivalent of
Wall Street.  I considered it a reasonably enjoyable existence.
 But then came the stock market crash . . .

I recently introduced a group of risk-based market metrics that
support a trading method that I call "Skipjack" - named after the
tuna.  A chart of a good skipjack might actually resemble a tuna.
 While the simulations running on an application that I call
Simtactics suggest that the methodology is often and perhaps
generally applicable, sometimes it is quite challenging to beat the
market on a total return basis.  This caused me to use the system
on large amounts of market data to ascertain the reasons.  However,
I needed a market player more consistent than myself in order to
remove irregularities extending from my unique personal qualities.
 I worked on an autopilot feature for Simtactics that eventually
led to the Levi-Tate Group of Market Metrics.  Yes, I own the
domain names.

The Levi-Tate metrics make it possible to produce a type of
topographical or algomorphological chart that I call a Thunderbird
Chart.  A form of this chart that I call an Advantage Chart follows
the extent to which the autopilot is able to beat the market.  I
found that the chart changes significantly over time.  I therefore
decided to follow the advantage levels over different time slices
about equidistant in terms of their periodicity.  At this point, it
became possible to study how the array or gradient of algorithmic
combinations respond to stock market conditions over time.  This
leads me to the special charts on this blog that allow me to
examine the progression of the algomorphology in a truly
developmental sense.  At this level, I confess the charts are a bit
complicated both to explain and also to present.  A 12x12 grid
containing 144 combinations from 1 to 56 results in 144 rows per
column of time slice.  Rather than dwell on difficulties
visualizing the exact details, please consider taking my word for
it in relation to the general details that I am about to

On the chart below, there is a 0 on the y-axis.  Above the 0,
the Simtactics trigger package was able to successfully beat market
using a particular combination.  Below 0, the application's
apparatus failed to beat market.  The chart below makes use of data
from the Dow Jones Industrial Average one year before the Crash of
1987.  Notice that a fair number of algorithmic pairs are above 0,
which means that they successfully beat market.  As I mentioned
earlier, it isn't unusual for these metrics to beat the market.  We
know however that quite a horrific crash occurred in 1987.  I will
therefore check out the chart immediately before the crash.

This being a method of technical analysis, it is strictly based
on pricing.  I do quite a lot with the pricing, of course.
 (Technical analysis also allows for the use of the volume, too.)
 Below is the chart immediately before the crash.  Sometimes when a
crash occurs, the exact dates are unclear.  I know in a sense it
should be clear - like an automobile crash.  But in the case of the
stock market, the crash might develop over a period of time.  In
any event, many people associate October 16, 1987 with the date of
the crash.  The chart below stops at October 15, 1987: this is
because my objective is to study the technical characteristics
before the crash to determine if they can predict what is about to
occur.  Notice how this chart is quite different from 1986: most of
the combinations led to outcomes below the 0.  I point out also
that the pattern began to splinter after October 5, 1987 (before
the crash).  The takeaway here is that before a major correction,
the algorithmic outcomes generally fall below 0.  More than 90
percent of the algorithmic pairs failed to beat market.  Many of
Simtactics' algorithmic traders were noticeably more successful by
October 15, 1987.

I added the next chart just to simplify my message since I
realize that the one immediately above presents information in a
fairly complicated manner.  The bar chart below shows that the
algorithmic failure rate was extremely high for some time before
the Crash of 1987.  But just before the crash, the artificial
traders were becoming much more successful beating the market.  I
suggest actually that a "normal" condition of risk diversity was
being restored to an otherwise highly polarized market.  There are
cases of humans remaining in dangerous, toxic, and unhealthy
conditions by rationalizing out the perceived risks.  After all,
this type of insulation is possible using alcohol or drugs.  But
certainly there are social mechanisms that can lead people to hold
beyond their natural risk tolerance levels.  Another idea that I
think has some merit is how survivorship might reward the risk
tolerant or condition those whose tolerance levels are adaptive or
elastic.  Many people invest by proxy these days - that is to say,
through professional portfolio managers who for their part might
make use of algorithmic traders.  There is therefore a level of
disassociation built into the system these days separating many
investors from their investments.  But as I mentioned in my
previous blog, this simply means that human stress perceptions have
been replaced by algorithmic risk perceptions.  I believe that
humans are capable of much more insulated thinking than
professionally designed algorithms.

This next chart is from the dot-com correction.  For those that
do not recall what caused this correction, essentially many
internet companies had unproven business models; nonetheless,
investors kept trading up the value of the stocks.  It is difficult
to determine the extent to which a stock might be overpriced in a
new market.  The chart below shows that for the most part, my
algorithmic pairs could not beat the market - until of course the
market collapsed in a drunken stupor.  The algorithms started to
gain steam all at the same time to the right of the chart.  So
there is some evidence that a widespread failure rate is itself an
indication of a market that seems more likely to encounter a large
correction.  However, within this context of failure as the chart
below shows, there are noticeable systematic developments that
appear to almost suggest a reversal in performance en route to the

Again I provide a bar chart as simplification.  It shows a very
high failure rate persisting before the market buckled.  By the
way, for those interested, skipjacking remains possible even if the
autopilot appears to be failing.  But I would say that there are
fewer clear opportunities for the player to win in a relative
sense; certainly it is much more difficult to beat the market.  On
the other hand, for those that assume the market position, they are
destined to obtain the market return both positive and

No study of stock market crashes would be complete without
consider the crash associated with the Great Depression.  I had my
fingers crossed - and there it is again!  Nearly all of the
algorithmic pairs performed below market immediately before the
crash.  The Crash of 1929 is one of those slow-motion horror
movies.  The dates on the x-axis indicate my purely technical
perspective on when the crash occurred.  Those that know their data
really well might point out that the my algorithmic pairs seem to
start advancing well before the crash.  Exactly.  The sudden
increase in effectiveness seems to provide some warning.

This leads me to my final chart which isn't about a crash -
given that most of the lines are above 0.  It is actually from
trading activity last Friday for a particular ETF.  While most of
the lines are beating market, consider the suspicious downward
pattern to the right of the chart.  By the way, this downward
motion does not necessarily mean that the price is declining.  It
means that the algorithms are becoming less successful than market.
 This poses an interesting question: would it be wise to take a
buy-and-hold strategy?  What I can say is that if a buy-and-hold
strategy is considered, it would probably be worthwhile to watch
for that splintering behaviour that seems to precede stock market

GPT3 and AGI: Beyond the dichotomy – part two

This blog continues from
GPT3 and AGI: Beyond the dichotomy – part one

GPT3 and AGI

Let’s first clarify what AGI should look like

Consider the movie ‘Terminator’

When the Arnold Schwarzenegger character comes to earth – he
is fully functional. To do so, he must be aware of the context. In
other words, AGI should be able to operate in any

Such an entity does not exist

And nor is GPT3 such an entity

But GPT3 however has the capacity to respond ‘AGI-like’ to
an expanded set of contexts much more than traditional AI

GPT 3 has got many things going for it

  • Unsupervised learning is the future
  • Linguistic capabilities distinguish humans
  • But Language is much more than encoding information. At a
    social level, language involvesjoint attention to environment,
    expectations and patterns.
  • Attention serves as a foundation for social trust
  • Hence, AGI needs a linguistic basis – but that needs
    attention and attention needs context. So, GPT-3 – linguistic –
    attention – context could lead to AGI-like behaviour

Does AGI need to be conscious as we know it or would
access consciousness suffice?

In this context, a recent paper

A Roadmap for Artificial General Intelligence: Intelligence,
Knowledge, and Consciousness: Garrett Mindt and Carlos

makes an argument is that

  • integrated information in the form of attention suffices for
  • AGI must be understood in terms of epistemic agency, (epistemic
    = relating to knowledge or the study of knowledge) and
  • Eepistemic agency necessitates access consciousness.
  • access consciousness: acquiring knowledge for action,
    decision-making,  and  thought,  without  necessarily  being 


Therefore, the proposal goes that AGI necessitates 

  • selective attention for accessing information relevant to
    action,  decision-making,  memory  and    
  • But not necessarily consciousness as we know it


This line of thinking leads to many questions

  • Is consciousness necessary for AGI?
  • If so, should that consciousness be the same as human
  • Intelligence is typically understood in terms of
    problem-solving. Problem solving by definition leads to specialized
    mode of evaluation. Such tests are easy to formulate but check for
    compartmentalized competency (which cannot be called intelligence).
    They also do not allow intelligence to ‘spill over’ from one
    domain to another – as it does in human intelligence. 
  • Intelligence needs information to be processed in a
    contextually relevant way.
  • Can we use epistemic  agency  through  attention as the
    distinctive mark of general intelligence even without
    consciousness? (as per Garrett Mindt and Carlos Montemayor)
  • In this model, AGI is based on joint attention to preferences
    in a context sensitive way.
  • Would AI be a peer or subservient in the joint attention

Finally, let us consider the question of spillover of
intelligence. In my view, that is another characteristic of AGI.
Its not easy to quantify because tests are specific to problem
types currently. A recent example of spillover of intelligence is
from facebook AI supposedly inventing it’s own secret
The media would have you believe that groups of
AGI are secretly plotting to take over humanity. But the reality is
a bit mundane as explained.
The truth behind facebook AI inventing a new language

In a nutshell, the system was using Reinforcement learning.
Facebook was trying to create a robot that could negotiate. To do
this, facebook let two instances of the robot negotiate with each
other – and learn from each other. The only measure of their
success was how well they transacted objects. The only rule to
follow was to put words on the screen. As long as they were
optimizing the goal(negotiating) and understood each other it did
not matter that the language was accurate (or indeed was English).
Hence, the news about ‘inventing a new language’. But
to me, the real question is: does it represent intelligence

Much of future AI could be in that direction.

To Conclude

We are left with some key questions:  

  • Does AGI need consciousness or access consciousness?
  • What is role of language in intelligence?
  • GPT3 has reopened the discussion but still hype and dichotomy
    (both don’t help because hype misdirects discussion and dichotomy
    shuts down discussion)
  • Does the ‘Bitter lesson’ apply? If so, what are its
  • Will AGI see a take-off point like Google translate did?
  • What is the future of bias reduction other than what we see
  • Can bias reduction improve human insight and hence improve
    Joint attention?
  • GPT-3 – linguistic – attention – context
  • If context is the key, what other ways can be to include
  • Does problem solving compartmentalize intelligence?
  • Are we comfortable with the ‘spillover’ of intelligence in
    AI? – like in the facebook experiment




image source: Learn English Words: