Data Warehouse Architectures

Data warehouses and their architectures vary depending upon the specifics of an organization's situation.

Data Warehouse Architecture (Basic)

shows a simple architecture for a data warehouse. End users directly access data derived from several source systems through the data warehouse.

Architecture of a Data Warehouse

This illustrates three things:

• Data Sources (operational systems and flat files)
• Warehouse (metadata, summary data, and raw data)
• Users (analysis, reporting, and mining)

The metadata and raw data of a traditional OLTP system is present, as is an additional type of data, summary data. Summaries are very valuable in data warehouses because they pre-compute long operations in advance. For example, a typical data warehouse query is to retrieve something like August sales. A summary in Oracle is called a materialized view.

Data Warehouse Architecture (with a Staging Area)

You need to clean and process your operational data before putting it into the warehouse. You can do this programmatically although most data warehouses use a Staging area instead. A staging area simplifies building summaries and general warehouse management.

Architecture of a Data Warehouse with a Staging Area

This illustrates four things:

• Data Sources (operational systems and flat files)
• Staging Area (where data sources go before the warehouse)
• Warehouse (metadata, summary data, and raw data)
• Users (analysis, reporting, and mining)

Data Warehouse Architecture (with a Staging Area and Data Marts)

Although the architecture in is quite common, you may want to customize your warehouse's architecture for different groups within your organization. You can do this by adding data marts, which are systems designed for a particular line of business.Illustrates an example where purchasing, sales, and inventories are separated. In this example, a financial analyst might want to analyze historical data for purchases and sales.

 Architecture of a Data Warehouse with a Staging Area and Data Marts

 


This illustrates five things:

• Data Sources (operational systems and flat files)
• Staging Area (where data sources go before the warehouse)
• Warehouse (metadata, summary data, and raw data)
• Data Marts (purchasing, sales, and inventory)
• Users (analysis, reporting, and mining)
  

Data Warehousing Concepts

The original concept of a data warehouse was devised by IBM as the ‘information ware house and presented as a solution for accessing data held in non-relational systems.

The abbreviated of DW is Data warehouse  is a relational database that is designed for query and analysis rather than for transaction processing is collection of data designed to support Management decision making.

 
A single, complete and consistent store of data obtained from a variety of different sources made available to end users in a what they can understand and use in a business context.

It usually contains historical data derived from transaction data, but it can include data from other sources. It separates analysis workload from transaction workload and enables an organization to consolidate data from several sources.

Development of a data warehouse includes development of systems to extract data from operating systems plus installation of a warehouse database system that provides managers flexible access to the data. 

Data warehouse: A subject-oriented, integrated, time-variant, and non-volatile collection warehousing tion of data in support of management’s decision-making process.

Subject-oriented: Integration is closely related to subject orientation. Data warehouses must put data from disparate sources into a consistent format (such as customers, products, and sales) rather than the major application areas (such as customer invoicing, stock control, and product sales)

Integrated : Integration is closely related to subject orientation. Data warehouses must put data from disparate sources into a consistent format. They must resolve such problems as naming conflicts and inconsistencies among units of measure. When they achieve this, they are said to be integrated.

The source data is often inconsistent using, for example (having different formats) because of different source data from different applications systems.

 

Time-variant: The data in the warehouse is only accurate and valid at some point in time or over some time interval

Non-Volatile: The data is not updated in real time but is refreshed from operational systems on a regular basis. New data is always added as a supplement to the database, rather than a replacement

Spiral Model

The spiral model is similar to the incremental model, with more emphasis placed on risk analysis. The spiral model has four phases: Planning, Risk Analysis, Engineering and Evaluation.

The spiral model combines the idea of iterative development with the systematic, controlled aspects of the waterfall model.

Spiral model is a combination of iterative development process model and sequential linear development model i.e. waterfall model with very high emphasis on risk analysis.

A software project repeatedly passes through these phases in iterations (called Spirals in this model). The baseline spiral, starting in the planning phase, requirements are gathered and risk is assessed. Each subsequent spirals builds on the baseline spiral. 

• High amount of risk analysis hence, avoidance of Risk is enhanced.
• Good for large and mission-critical projects.
• Strong approval and documentation control.

Software is produced in the engineering phase, along with testing at the end of the phase.  The evaluation phase allows the customer to evaluate the output of the project to date before the project continues to the next spiral.

Diagram of Spiral model:

Incremental Model

In incremental model the whole requirement is divided into various builds. Multiple development cycles take place here, making the life cycle a“multi-waterfall” cycle.  Cycles are divided up into smaller, more easily managed modules.  Each module passes through the requirements,

Incremental model is an evolution of waterfall model. Multiple development cycles take place here, making the life cycle a “multi-waterfall” cycle.  Cycles are divided up into smaller, more easily managed iterations.  Each iteration passes through the requirements, design, implementation and testing phases.

§  More flexible – less costly to change scope and requirements.

§  Easier to test and debug during a smaller iteration.

Diagram of Incremental model:

V-Model

V- model means Verification and Validation model. Just like the waterfall model, the V-Shaped life cycle is a sequential path of execution of processes. Each phase must be completed before the next phase begins.  Testing of the product is planned in parallel with a corresponding phase of development.

V - Model is an extension of the waterfall model and is based on association of a testing phase for each corresponding development stage. This means that for every single phase in the development cycle there is a directly associated testing phase. This is a highly disciplined model and next phase starts only after completion of the previous phase.

Diagram of V-model:

Verification Phases

Following are the Verification phases in V-Model:

• Business Requirement Analysis: This is the first phase in the development cycle where the product requirements are understood from the customer perspective. This phase involves detailed communication with the customer to understand his expectations and exact requirement. 

• System Design: Once you have the clear and detailed product requirements, it.s time to design the complete system. System design would comprise of understanding and detailing the complete hardware and communication setup for the product under development. System test plan is developed based on the system design. Doing this at an earlier stage leaves more time for actual test execution later.

• Architectural Design: Architectural specifications are understood and designed in this phase. Usually more than one technical approach is proposed and based on the technical and financial feasibility the final decision is taken. System design is broken down further into modules taking up different functionality. 
• The data transfer and communication between the internal modules and with the outside world (other systems) is clearly understood and defined in this stage. With this information, integration tests can be designed and documented during this stage.

• Module Design:In this phase the detailed internal design for all the system modules is specified, referred to as Low Level Design (LLD). It is important that the design is compatible with the other modules in the system architecture and the other external systems. Unit tests are an essential part of any development process and helps eliminate the maximum faults and errors at a very early stage. Unit tests can be designed at this stage based on the internal module designs.

Validation Phases

Following are the Validation phases in V-Model:

• Unit Testing: Unit tests designed in the module design phase are executed on the code during this validation phase. Unit testing is the testing at code level and helps eliminate bugs at an early stage, though all defects cannot be uncovered by unit testing.

• Integration Testing: Integration testing is associated with the architectural design phase. Integration tests are performed to test the coexistence and communication of the internal modules within the system.

• System Testing: System testing is directly associated with the System design phase. System tests check the entire system functionality and the communication of the system under development with external systems. Most of the software and hardware compatibility issues can be uncovered during system test execution.

• Acceptance Testing: Acceptance testing is associated with the business requirement analysis phase and involves testing the product in user environment. Acceptance tests uncover the compatibility issues with the other systems available in the user environment. It also discovers the non functional issues such as load and performance defects in the actual user environment.

Waterfall Model

The Waterfall Model was first Process Model to be introduced. It is also referred to as a linear-sequential life cycle model. It is very simple to understand and use. In a waterfall model, each phase must be completed fully before the next phase can begin.   At the end of each phase, a review takes place to determine if the project is on the right path and whether or not to continue or discard the project. In waterfall model phases do not overlap.

Waterfall model is the earliest SDLC approach that was used for software development .

The sequential phases in Waterfall model are:

• Requirement Gathering and analysis: All possible requirements of the system to be developed are captured in this phase and documented in a requirement specification doc.

• System Design: The requirement specifications from first phase are studied in this phase and system design is prepared. System Design helps in specifying hardware and system requirements and also helps in defining overall system architecture.

• Implementation: With inputs from system design, the system is first developed in small programs called units, which are integrated in the next phase. Each unit is developed and tested for its functionality which is referred to as Unit Testing.

• Integration and Testing: All the units developed in the implementation phase are integrated into a system after testing of each unit. Post integration the entire system is tested for any faults and failures.

• Deployment of system: Once the functional and non functional testing is done, the product is deployed in the customer environment or released into the market.

• Maintenance: There are some issues which come up in the client environment. To fix those issues patches are released. Also to enhance the product some better versions are released. Maintenance is done to deliver these changes in the customer environment.

Software Development Life Cycle (SDLC)

Here in this blog I will tell you what is software development life cycle,the phases of software development life cycle and Methods of SDLC.

What is Software Development Life Cycle?

SDLC, Software Development Life Cycle is a process used by software industry to design, develop and test high quality softwares. The SDLC aims to produce a high quality software that meets or exceeds customer expectations, reaches completion within times and cost estimates.

SDLC is a process followed for a software project, within a software organization. It consists of a detailed plan describing how to develop, maintain, replace and alter or enhance specific software. The life cycle defines a methodology for improving the quality of software and the overall development process.

There are various software development approaches defined and designed which are used/employed during development process of software, these approaches are also referred as
Software Development Process Models (SDLC MODELS) 
(e.g Waterfall model, V model, Incremental model, Spiral model)

Software life cycle models describe phases of the software cycle and the order in which those phases are executed. Each phase produces deliverables required by the next phase in the life cycle. Requirements are translated into design. Code is produced according to the design which is called development phase. After coding and development the testing verifies the deliverable of the implementation phase against requirements.

There are following six phases in every Software development life cycle model:

1. Requirement gathering and analysis
2. Design
3. Implementation or coding
4. Testing
5. Deployment
6. Maintenance

1) Requirement gathering and analysis
This phase is the main focus of the project managers and stake holders. Meetings with managers, stake holders and users are held in order to determine the requirements like; Who is going to use the system? How will they use the system?  What data should be input into the system?  What data should be output by the system?  These are general questions that get answered during a requirements gathering phase. After requirement gathering these requirements are analysing for their validity and the possibility of incorporating the requirements in the system to be development is also studied.

2) Design:  In this phase the system and software design is prepared from the requirement specifications which were studied in the first phase. System Design helps in specifying hardware and system requirements and also helps in defining overall system architecture. The system design specifications serve as input for the next phase of the model.

3) Implementation / Coding:  On receiving system design documents, the work is divided in modules/units and actual coding is started. Since, in this phase the code is produced so it is the main focus for the developer. This is the longest phase of the software development life cycle.

4)  Testing:  After the code is developed it is tested against the requirements to make sure that the product is actually solving the needs addressed and gathered during the requirements phase. During this phase unit testing, integration testing, system testing, acceptance testing are done.

5)  Deployment: After successful testing the product is delivered / deployed to the customer for their use.

6) Maintenance: Once when the customers starts using the developed system then the actual problems comes up and needs to be solved from time to time. This process where the care is taken for the developed product is known as maintenance.

What is the difference between JRE,JVM and JDK?

Java Development kit(JDK):

 

JDK is mainly targeted for java development. I.e. You can create a Java file (with the help of Java packages), compile a Java file and run a java file.

Java Developer Kit contains tools needed to develop the Java programs, and JRE to run the programs. The tools include compiler (javac.exe), Java application launcher (java.exe), Appletviewer, etc…

Compiler converts java code into byte code. Java application launcher opens a JRE, loads the class, and invokes its main method.

You need JDK, if at all you want to write your own programs, and to compile them.

For running java programs, JRE is sufficient. JRE is targeted for execution of Java files i.e. JRE = JVM + Java Packages Classes(like util, math, lang, awt,swing etc)+runtime libraries. JDK is mainly targeted for java development. I.e. You can create a Java file (with the help of Java packages), compile a Java file and run a java file. 

The Java Development Kit (JDK) is an Oracle Corporation product aimed at Java developers. Since the introduction of Java, it has been by far the most widely used Java Software Development Kit (SDK).

 

JRE (Java Runtime Environment)

Java Runtime Environment contains JVM, class libraries, and other supporting files. It does not contain any development tools such as compiler, debugger, etc. Actually JVM runs the program, and it uses the class libraries, and other supporting files provided in JRE. If you want to run any java program, you need to have JRE installed in the system

JVM (Java Virtual Machine)

The Java Virtual Machine provides a platform-independent way of executing code

JVM has to interpret the byte codes to machine language, then it has to use some native or operating system specific language to interact with the system. One has to be very clear on platform independent concept.

As we all aware when we compile a Java file, output is not an ‘exe’ but it’s a ‘.class’ file. ‘.class’ file consists of Java byte codes which are understandable by JVM. Java Virtual Machine interprets the byte code into the machine code depending upon the underlying operating system and hardware combination. It is responsible for all the things like garbage collection, array bounds checking, etc… JVM is platform dependent..

Click here to View all Programing languages

If you are a Java developer, it is very often that you think about understanding the JRE,JVM and JDK. Once if you understand these things, it would be quite easy for you to visualize things in logical manner. Also look at the picture below, that will clear all your questions about the JRE,JVM and JDK. This article explains about the each term and will make you understand perfectly - See more at: http://www.javabeat.net/2013/02/what-is-the-difference-between-jrejvm-and-jdk/#sthash.Pmr6lNmp.dpuf

If you are a Java developer, it is very often that you think about understanding the JRE,JVM and JDK. Once if you understand these things, it would be quite easy for you to visualize things in logical manner. Also look at the picture below, that will clear all your questions about the JRE,JVM and JDK. This article explains about the each term and will make you understand perfectly - See more at: http://www.javabeat.net/2013/02/what-is-the-difference-between-jrejvm-and-jdk/#sthash.Pmr6lNmp.dpuf

If you are a Java developer, it is very often that you think about understanding the JRE,JVM and JDK. Once if you understand these things, it would be quite easy for you to visualize things in logical manner. Also look at the picture below, that will clear all your questions about the JRE,JVM and JDK. This article explains about the each term and will make you understand perfectly - See more at: http://www.javabeat.net/2013/02/what-is-the-difference-between-jrejvm-and-jdk/#sthash.Pmr6lNmp.dpuf

If you are a Java developer, it is very often that you think about understanding the JRE,JVM and JDK. Once if you understand these things, it would be quite easy for you to visualize things in logical manner. Also look at the picture below, that will clear all your questions about the JRE,JVM and JDK. This article explains about the each term and will make you understand perfectly - See more at: http://www.javabeat.net/2013/02/what-is-the-difference-between-jrejvm-and-jdk/#sthash.Pmr6lNmp.dpuf