Data warehouse

Oracle Materialized views to optimize a Datawarehouse

carlos — Thu, 23 Oct 2008 13:13:20 +0200

Datawarehouse loadings are periodic, and summary tables are useful to improve efficiency and the answer time of our reports. A physical optimisation resource that can bring big improvements in our system is the utilization of materialized views.

The view materialized is not more that a view, defined with one sentences SQL, of which in addition to storing his definition, store the data that returns, realizing an initial load and afterwards each true time a freshen of the same.

Like this, if we have a Datawarehouse that update daily, could use views materialized to go updating intermediate tables that feed our diagrams of DWH, or directly to implement tables agregadas that will freshen from our tables base.

The creation of this type of views no so complex as it can seem, the most important is to have clear each how much time want to update the information of the views, and which method of freshen to use.

Also will have to ensure us that our licence of base of data allows us use them (has to be a version Enterprise).

Sintaxis Basic for the creation of a view materialized

CREATE MATERIALIZED VIEW My_view_materialized
             [TABLESPACE My_tablespace]
             [BUILDunknown{^*IMMEDIATE | DEFERRED}]
             [REFRESHunknown{^*ON COMMIT | ON DEMAND | [START WITH Dates_start] NEXT dates_interval } |.Verbcj | FAST | FORCE} ]
             [unknown{^*ENABLE|DISABLE} QUERY REWRITE]
AS SELECT t1.Field1, t2.Field2
      FROM My_table1 t1 , my_table2 t2
      WHERE t1.Field_fk = t2.Field_pk
         AND …

Comments on the different options:

Load of data in the view

BUILD IMMEDIATE:
The data of the view load in the same moment of the creation

BUILD DEFERRED:
Only create the definition, the data will load more forward. To realize this load can use the function REFRESH of the package DBMS_MVIEW:
      begin
         dbms_mview.refresh('My_view_materialized');
      end;

Method and temporalidad of the freshen of the data

Each how much time will freshen :

REFRESH ON COMMIT:
Each time that do a commit in the objects origin defined in the select

REFRESH ON DEMAND:
As with the option DEFERRED of the BUILD, will use the procedures REFRESH, REFRESH_ALL_MVIEWS or REFRESH_DEPENDENT of the package DBMS_MVIEW

REFRESH [START WITH Dates_start] NEXT dates_interval:
START WITH Indicate the date of the first freshen (dates_start suele be a SYSDATE)
NEXT indicates each how much time will update (dates_interval could be SYSDATE +1 to realize the freshen once to the day)

Of which way will freshen

REFRESH COMPLETE:
The freshen will do of all the data of the view materialized, the recreará entirely each time that launch the freshen

REFRESH FAST:
The freshen will be incremental, is the option more recomendable, the of fast already gives an idea of the porqué.
This type of freshen has quite a lot of restrictions as the type of view that was creating .
They can consult generally Restrictions on Fast Refresh of the official documentation of Oracle

One of the important things to have in account is that to can use this method almost always is necessary to have created before a LOG of the View materialized, indicating the key fields in which will base the maintenance of the view.
It uses the instruction CREATE MATERIALIZED VIEW LOG ON:

   CREATE MATERIALIZED VIEW LOG ON My_table_origin
      WITH PRIMARY KEY
      INCLUDING NEW VALUES;

REFRESH FORCE:
With this method indicate that if it is possible use the metodo FAST, and if no the COMPLETE.

To know if a view materialized can use the method FAST, the package DBMS_MVIEW provide the procedure EXPLAIN_MVIEW

Activación Of the reescritura of queries

ENABLE QUERY REWRITE:
Allow to the base of data the reescritura of queries

DISABLE QUERY REWRITE:
desactiva The reescritura of queries

The option QUERY REWRITE is the one who more go to use if we love the views materialized to optimize ours Date warehouse.
This option allows to create tables agregadas in form of views materialized, and that when it launch a SELECT the base of data can reescribirla to consult the table or view that go to give back the data requested in less time, everything of way totally transparent to the user

The only that there is that do is to create the tables agregadas like views materialized with QUERY REWRITE enabled.

Examples of views materialized

They are a lot of combinations, but the final sentence is not so complex.

First step of the ETL

If we wanted to create a view materialized of a table that freshen a day to the week, and of way incremental would do the following:

CREATE MATERIALIZED VIEW LOG ON My_table_origin
   WITH PRIMARY KEY
   INCLUDING NEW VALUES;
CREATE MATERIALIZED VIEW My_view_materialized
REFRESH FAST NEXT SYSDATE + 7
AS SELECT field1, field2, field8
        FROM My_table_origin
    WHERE Field2 > 5000;

This view could serve us to feed the load of a Dates Mart that it realized weekly. We could programarla so that it freshened fair before the start of the process of load, or as first step in the ETL, and already would have the necessary data updated, and independent of the origin of data (would not have to bother more to the operacional). Another advantage to have in account is that if there is some problem with the access to the data origin, if we have not deleted them, in the view materialized still will have the data of the last freshen, with what although the freshen failed would not find us an error that truncated the load of ours Date Warehouse, or an empty table.
Of course, in the conditions of the WHERE could select only the necessary registers, only the ones of the last month, etc.

Tables agregadas

Another important example would be the utilization of views materialized for the creation of tables agregadas:

CREATE MATERIALIZED VIEW Sales_agregadas_mv
BUILD IMMEDIATE
REFRESH COMPLETE
ENABLE QUERY REWRITE
AS
SELECT Go_product, sum(amount) total_sales
FROM Sales;

With this simple sentence would create a table agregada of total of sales by product of a supposed table of sales than serious the table of facts.

To level of session also there would be that ensure that the option QUERY_REWRITE was actuated. Just in case enable with

ALTER SESSION SET QUERY_REWRITE_ENABLED=TRUE;

If now inside this session executes the sentence

SELECT sum(Amount)
FROM Sales;

The base of data will prepare the plan of execution having in account the view materialized created and internally will realize the selection on the seen sales_agregadas_mv.

A simple way to check it, apart to examine the plan of execution, or to compare time before and after the creation of the view, or desactivando the QUERY_REWRITE, is to check that this sentence gives back resulted in the same time that the sentence

SELECT sum(Amount)
FROM Sales_agregadas_mv;

To consult more details, or the sintaxis complete of the creation of views materialized in the chapter Create Materialized View of the manual of reference SQL of Oracle

With enterprise manager or with the console web of the base of data also can create the views materialized of a more assisted way, but equally is important to have clear the concepts before doing it.

Virtual image with Pentaho configured

carlos — Thu, 23 Oct 2008 12:58:58 +0200

We have prepared a virtual image with an installation preconfigurada of Pentaho, that can discharge freely from the site LinuxBI.

The image works with the software of virtualización Sun VirtualBox 2.0. We have chosen Virtual Box because it is available for Linux, OS X (Mac) and Windows, and because dispone of a free edition of evaluation and personal use, and another Open Source subject to the conditions of the licence GPL. Of this way our image will be able to use freely in any surroundings, always that they respect the conditions of the licences.

Virtualización

VirtualBox Install with ease in any surroundings, and his operation is very intuitive, but attach the manual of user for the one who do not know the tool, or want to use options advanced.
To use our servidor virtual can follow these simple steps:

Install VirtualBox
Discharge the virtual disk from virtual Image Pentaho
Descomprimirlo In the directory of virtual disks of VirtualBox, usually call VDI
Execute VirtualBox and create a new machine with the assistant of the application, choosing the type of Operative System Net Hat, 512 Mb by heart (this is a suggestion), and the virtual disk discharged like disk of start.
Arrancar The machine doing double click on her.

Like Operative System base for the installation of Pentaho has chosen CentOS 5. The reasons also have to see with the use that can him give to this image. This UNDER Linux is based in the code free source that uses Net Hat Enterprise Linux and, to practical effects, works of the same way that this known business version of Linux, and is orientado also to a business use, but is totally of free distribution. Can use until in some surroundings of production, and most of the documentation and practical of RedHat are applicable to CentOS.

It has created the user dataprix. The password so much of this user and of the user root is also dataprix.

Agradecemos To Oscar Osta the preparation of virtual image initial with the installation of CentOS.

Pentaho

It has done an installation of Pentaho BI Suite PCI, that is the installation of Pentaho that comes preconfigurada and with a base of data Hypersonic (HSQLDB) operative and with data of proof so that they can execute the reports and 'applications' that can find in the portal of example that have accessible when raising the servidor.

Can use this installation to test the platform in your proper machine/servidor, exploring the options of the portal, or even can use like base to build a proper system of BI with Pentaho.

The last instantaneous that has captured of the state of the virtual machine already has the servidor of BI raised and the open portal in the navigator Firefox. The users of the portal already have the password entered.

aneras, indicate how can raise the servidor each time that start the machine:

Once created the virtual machine, the steps to go in in the session and raise the servidor of BI are the following

Arrancar The virtual machine from the interface graphic of VirtualBox
Go in with the user dataprix, password dataprix
Do double click in the Terminal Icon of the desk
In the window of the terminal execute the following commandos:
$ cd pentaho-demo (Directory of the demo)
$ ./start-pentaho.sh (shell Of start of the platform)
Expect to that in the terminal appear the message Pentaho BI server smart
Open the browser Firefox (icon to the side of the option of menú System)
The url of access to the portal is http://localhost:8080/pentaho

Each time that reinicie the machine there be that execute this script if you want to use the portal, also could do that it executed the shell each time that initiate the servidor.

To detain the servidor Pentaho, opening another terminal:

$ cd pentaho-demo
$ ./stop-pentaho.sh

This is the most basic installation, that is configured to work in venue and on Hypersonic, perfect to evaluate the possibilities of this software open source of Business Intelligence, or for a first takes of contact without a lot of complications. Also serves like base to prepare things more complex, and expect in a future can go broadening this virtual image with new functionalities, configurations, proofs or give interesting.

If somebody uses this image like base to do something interesting agradeceremos that us explain it, or that put in contact with us if it loves that we lodge his 'version'. Equally any comment, doubt, idea or suggestion will be bienvenido in our forum, in the subject Virtual Image with Pentaho.

BI book: Business Intelligence, Compete with Information

carlos — Mon, 22 Sep 2008 15:17:22 +0200

Josep Lluis Cano is the author of the book Business Intelligence: compete with Information. This excellent work explains in what consists the Business Intelligence and can be a big guide for abordar projects of BI. It is orientado sobretodo for directors of pymes that want to go in in the world BI, and do his more competitive company, by what the lenguaje is clear and the very practical content.

The first chapters explain all the necessary to know what is the BI, what can aportar to the company, which are his technical bases, and how can organize a project of Business Intelligence:

Chapter 1. Introduction to the Business Intelligence
Chapter 2. Modelización Of the Business
Chapter 3. Model of data
Chapter 4. Components of Business Intelligence
Chapter 5. Projects of Business Intelligence

The two last chapters are especially practical, and contain references to different tools and providers of BI and cases of success of implementations of projects of this type:

Chapter 6. Selection of tools and providers
Chapter 7. Experiences of implementation of Business Intelligence

The last chapter throws a look to the current market of BI, and comment to where direct :

Chapter 8. Uses of Business Intelligence and new trends

At the end include also an annex on the Model Relacional of bases of data and the lenguaje SQL and a complete bibliographic reference and webgráfica.

The electronic book in format pdf can discharge of Business Intelligence: compete with Information, and has been obtained of ScriptBD, where specify that it is under a licence of attribution no commercial

Oracle Warehouse Builder 10g general availability

carlos — Tue, 27 May 2008 00:19:49 +0200

If you work with Oracle databases and are thinking about using and ETL (Extract, Transform and Load) tool to create or support a Data warehouse system, or simply to facilitate data integration or migrations, you have to know that Oracle allows to use his Oracle Warehouse Builder basic version without any license charge. The only requirement is that you have at least a license of Oracle Database Standard Edition One, Oracle Database Standard Edition or Oracle Database Enterprise Edition.

More details at Oracle Announces General Availability of Oracle(r) Warehouse Builder 10g Release 2