Ab Initio - Session6

AB INITIO EME
ABINITIO
AB INITIOCONFIDENTIAL
CONFIDENTIAL AND
AND PROPRIETARY
PROPRIETARY 1
AB INITIO EME
The
Enterprise
Meta>Enviro
nment
for Ab Initio Developers
AB INITIO CONFIDENTIAL AND PROPRIETARY 2

AB INITIO EME Confidential and
Proprietary!
All software and training material is:

Copyright 1994-2002
Ab Initio Software Corporation
Presentations, on-line files, and printed matter
are covered by nondisclosure agreement(s).
Course material and documentation are not to
be circulated to organizations or individuals
not under nondisclosure.

AB INITIO EME
Outline
Overview: What is the Enterprise
Meta>Environment?
Using the EME...

...from the GDE
...from the shell
...from the web and GDE
...from the shell

AB INITIO EME
Overview

AB INITIO EME
Overview
What is the
Enterprise Meta>Environment?

AB INITIO EME
What is an EME?
meta>environment:
an environment for metadata

AB INITIO EME
What is metadata?
Data about data.
Record formats, database ddl, field/column

definitions, business rules, business processes,
job statistics, version history, etc.
Everything but the data itself.

AB INITIO EME
What is an EME?
meta>environment:
organization
management
use

AB INITIO EME
What is an EME?
meta>environment:
organization storage
management manipulation
use analysis

AB INITIO EME
Two classes of metadata
Ab Initio-related
graphs, dml, xfr, db, etc.
= Technical metadata
Everything else
= Business metadata

AB INITIO EME Organize by
classification: project
Separate technical metadata

ACC
by type
db
db - database-related
dml - record formats
dml
mp - graphs mp
run - deployed scripts run
xfr - transforms xfr

AB INITIO EME Organize by subdivision:
application
Divide technical metadata EDW
into projects
ACC CUS SHD_EDW
conceptually
for logical coherence
organizationally
for manageability

AB INITIO EME Technical metadata EME
uses
Central Metadata Storage

Versioning
Source code control
Analysis
Job status

AB INITIO EME
More about the EME...
from the developers

perspective

AB INITIO EME
Metadata for projects
project metadata:
parameters
name-value pair
logical-physical mapping
AI_DML $PROJECT_DIR/dml
AI_RUN $PROJECT_DIR/run
AI_XFR $PROJECT_DIR/xfr

AB INITIO EME
Common Projects
To use another projects parameters:

include it as a common project
/Projects/stdenv Common Project /usr/admin/sand/stdenv

/Projects/pub_edw Common Project /usr/admin/sand/pub_edw
AI_DML String $PROJECT_DIR/dml

AB INITIO EME
Public vs private projects
Any project can be included

but not all should be!
public projects
intended to be included
private projects
not to be included

AB INITIO EME
Core concept: sandbox
A sandbox is a users personal

workspace:
Graphs and metadata are checked out
from the EME into a sandbox.
New and modified graphs and metadata
are then checked in to the EME.

AB INITIO EME
Sandbox Structure
Metadata separated Referring

ACC
Parameters
by type
$AI_DB
db
dml - record formats dml $AI_DML
mp - graphs mp $AI_MP
run - deployed scripts run $AI_RUN
xfr - transforms xfr $AI_XFR

AB INITIO EME Sandbox-Project
correspondence
Sandbox is a file system copy of a EME

project.
analogs
Sandbox1
Project Sandbox2
EME File
system
AB INITIO EME
Users
Each user
has their own sandbox(es)
one to a project
Project may have multiple copies in file

system.

AB INITIO EME Sandbox-project
correspondence
Sandboxes are distinguished from

projects.
They may contain objects that are either

not checked in to source control
currently being modified
the same as in project

AB INITIO EME EME-File system
correspondence
Directory parameters establish map

between file system and EME
$AI_DML
dml dml
EME File
system
AB INITIO EME
Using the EME
from the GDE

AB INITIO EME
GDE uses of the EME
Developers - Source Code Control:

Check Out
Check In
Analyze Dependencies
Administration
Create Projects
Edit Projects

AB INITIO EME
Configuring the gde

AB INITIO EME
Project>Settings
The Project>Settings dialog:
Click the Connect button.

Save your settings as Normal.aip.
AB INITIO EME Configuring
Run>Settings
Set your Run>Settings to Normal
Fill in host, login, and password
Leave Host directory blank

AB INITIO EME
Checking Out
Graph
Repository
Sandbox

AB INITIO EME
Why?
Check Out
to look at (latest) version
before modifying
Best Practice: Check Out

project at a time

AB INITIO EME
File system-EME
The EME stores objects under source

control.
Your work area is called a sandbox.
Check Out objects from the EME to your

sandbox.

AB INITIO EME
Creating a sandbox
Click Projects>Check Out

EME project
Host profile
Sandbox directory
Browse for a sandbox directory

Next>. Check Details. Do Checkout.

AB INITIO EME Checking out individual
objects
Check Out applies to graphs and other

objects as well.
During Check Out, simply browse to the

object you want to checkout.

AB INITIO EME
Modifying a graph
Client

AB INITIO EME Demonstration/Participati
on
Modify the graph
EDW_populate_from_catalog_cust_info.mp
Check In your changes.

AB INITIO EME
Core concept: locks
When a graph is first checked out from

the EME, it is in Read-only mode.
lock
To edit the graph, one must lock it so
that others cannot edit it at the same
time.

AB INITIO EME
Core concept: Locking
Lock objects before modifying them.
You cant lock if someone else already

has.
Locking controls access.

AB INITIO EME
Conflict management
No branching or merging.
Simultaneous editing blocked.
Lock breakable by administrator.

AB INITIO EME
Exercise.
Modify a graph you have opened, e.g.

AB INITIO EME
Checking In
Graph
Repository
Sandbox

AB INITIO EME
Why?
Check In
for source code control
to share code
Best Practice: Check In

one graph at a time

AB INITIO EME
Project>Check In
Host profile
Filename
Project

AB INITIO EME
Exercise.
Modify a graph, e.g.
and Check In.

AB INITIO EME
Label versions with tags
Assign tags to label version of an object.
Enables
differencing
reverting
promoting

AB INITIO EME
Check In with tags
Click here
to set tag.

AB INITIO EME
Tagging
Tag page
Set tag here
Add comment here

AB INITIO EME Tags apply to related
objects
When you tag a graph, all its related dml

and xfr files are also tagged.
Apply the same tag to more than one

graph to identify a set.
An object can have more than one tag.

AB INITIO EME
Configuration
You can apply a tag to a set of tags to

create a configuration.
Configurations identify objects as part

of a release for promotion.

AB INITIO EME Check In files and
projects
Check In applies to individual files, as well

as whole projects.
Simply browse to the appropriate object

during Check In.

AB INITIO EME
Check In is selective
Check In only applies to objects that have

been modified.
Its okay to Check In a graph even if only

a dml file has changed.

AB INITIO EME
Dependency Analysis Warnings

AB INITIO EME Core concept:
dependency analysis
The EME understands graphs.
Dependency analysis builds connections

between objects in the EME.
This enables the EME to deduce the flow

of data between graphs.

AB INITIO EME Parameters and
dependency analysis
Parameters for directories are used to

construct a mapping between metadata
locations in a sandbox and locations in
the EME.
This is the key to dependency analysis.
Warnings result when the GDE cannot
identify the mapping because a
parameter is not defined.
AB INITIO EME Dependency analysis:
warnings
Undefined parameter

AB INITIO EME
Warnings and resolution
At Check In, warnings issued

for undefined parameters
for invalid dml or xfrs
Resolve by
defining parameters
fixing errors

AB INITIO EME
Sandboxes

AB INITIO EME
Core concept: sandbox
A sandbox is a users personal

workspace:
Graphs and metadata are checked out
from the EME into a sandbox.
New and modified graphs and metadata
are then checked in to the EME.

AB INITIO EME
Sandbox Structure

ACC
Parameters
by type
$AI_DB
db

AB INITIO EME Sandbox-Project
correspondence
Sandbox is a file system copy of a EME

project.
analogs
Sandbox1
Project Sandbox2
EME File
system
AB INITIO EME
Users
Each user
has own sandbox(es)
one to a project
Project may have multiple copies in file

system.

AB INITIO EME
Access control
To view graphs in GDE,

you must Check Out to a sandbox.
Permissions
allow read-only access
block access

AB INITIO EME
Exercise. New Graphs
Create a new graph with File>New.

The Graph is not locked but is editable
since it is not known to the EME.
The GDE does not yet associate your
graph with a sandbox.
Save it to a sandbox to allow the
parameters to be used.

AB INITIO EME
Using parameters
HOUSEHOLD $AI_MFS/household.dat
INDIVIDUAL $AI_MFS/individual.dat
STORE_ACCOUNT $AI_MFS/store_account.dat

AB INITIO EME Demonstration/Participati
on
Look at the sandbox parameters of
/Projects/basic/cus.
Add a parameter REPORT with value

$AI_MFS/report.dat.

AB INITIO EME Core concept: Project
Parameters
Every Ab Initio project uses logical names

for physical values that may change
between runs
between project lifecycle stages
between machines.

AB INITIO EME
Parameters
Use parameters
to enhance portability
to promote flexibility
to simplify communication

AB INITIO EME
Examples
Examples:
Metadata directories (dml, xfr)
Data directory names (serial or MFS)
Database Names and install locations

AB INITIO EME
Sandbox Structure

ACC
Parameters
by type
$AI_DB
db

AB INITIO EME
Editing parameters
Use Project>Edit Sandbox>Parameters to

access parameters
Most parameters have

scope = local
type = string

AB INITIO EME
View>Columns

AB INITIO EME EME vs. Sandbox
parameter values
Parameters values may differ between

the EME and sandbox.
Example: PROJECT_DIR is root of project

Sandbox: PROJECT_DIR /users/jsmith/lesson
EME: PROJECT_DIR /Projects/lesson

AB INITIO EME
Example. Sandbox value
Example: Sandbox value

Directory path,
e.g. AI_DML for $PROJECT_DIR/dml
may resolve to
/users/jsmith/lesson/dml

AB INITIO EME
Example. EME value
Example: EME value

Directory path,
e.g. DML for $PROJECT_DIR/dml
may resolve to
/Projects/lesson/dml

AB INITIO EME
Common projects
Include another project to use its

parameters.
The included project is called a

common project.
Any project can be common.

AB INITIO EME
No circular references
No circular references CUS CRM
Instead share SHD_EDW
a common project.
CUS CRM

AB INITIO EME
Sharing metadata
Put shared metadata SHD_EDW
in a project that
other projects include. CUS CRM
Datasets that are shared

must be defined in
included projects.
AB INITIO EME

AB INITIO EME
Using the EME
...from the shell

AB INITIO EME
Invoking the environment
Two ways: In a sandbox

. ab_setup_project.ksh $PWD
or:
. <path to stdenv sandbox>/.project-start.ksh
set_environment $PWD

AB INITIO EME Core concept: air
commands
Interact with the EME from the shell using

air commands
Classes of commands
repository
project
sandbox
object

AB INITIO EME
AB_AIR_ROOT
Specify the default EME with the

environment variable
AB_AIR_ROOT
For example,
export AB_AIR_ROOT=$PWD/repo_training

AB INITIO EME
Basic commands
EME listing:
air ls -l <rpath>
Information about a command

air help <command>
For example,
air help ls
AB INITIO EME
air repository commands

AB INITIO EME
Create and remove
Create a new EME

air repository create <name>
Remove an EME
air repository destroy

AB INITIO EME
Backup commands
air repository backup allows any backup

command to be run on a locked EME
air repository create-image saves an
entire EME in a portable text format
air repository load-from-image creates a
EME from a saved image file

AB INITIO EME
air project, sandbox and object

commands

AB INITIO EME
Displaying information
Project information
air project show <project-name>
Parameter evaluation
air project parameter <proj> -eval <parm>

AB INITIO EME
air project show
Directories that are scanned

during Check In (import) --
** means "and all subdirectories".
Specify where non-project-related

items are stored.
Extension list that controls how

files are imported into a project:
ignore means "do not check in".

AB INITIO EME
Check In/Check Out
Check In
air project import <proj> -basedir <root>
Check Out
air project export <proj> -basedir <root>
Many other options.

AB INITIO EME Analysis Maintenance
commands
air project show-queue shows contents of

queues
air project dequeue removes graphs from
the dependency analysis or translation
queues
air project enqueue adds graphs to the
dependency analysis or translation
queues
AB INITIO EME Sandbox Maintenance
commands
air sandbox create creates a new sandbox

air sandbox parameter edits and views sandbox
parameters
air sandbox lock locks a file in a sandbox
air sandbox revert unlocks a file in a sandbox,
and restores it to the latest checked in version
air sandbox detach uncouples a sandbox from an
EME project

AB INITIO EME
Promotion
Save to a portable format

air save <file> <rpath> <>
Load from a portable format

air load <file>

AB INITIO EME General object
commands
air mv move an object

air cp copy an object
air ln create a symbolic link
air rm remove an object

AB INITIO EME
Tagging

AB INITIO EME
What is a Tag?
A "tag" is a label which marks a set of

specific versions of repository objects.

AB INITIO EME
Using the EME
for configuration
management

AB INITIO EME
Core concept: Promotion
Promotion refers to the movement of

graphs and associated code from one
environment to another.
The EME automates much of promotion

and mitigates many of the risks.

AB INITIO EME
Promotion Features
Primary objects (usually graphs) and

associated secondary objects (DML,
XFR, shell scripts, etc) move together.
Ensures that errors are not introduced

into applications as a result of
promotion

AB INITIO EME
Promotion
A standard promotion path

Development Testing Production
Promotion Promotion
EME EME EME
Check in
Check out
Check out
Check out
EME
Sandboxes on the file system
Developer Tester Production

Sandboxes Sandbox Sandbox
in File System in File System in File System

AB INITIO EME Developer View of
Promotion
Developers
checkin and checkout code
run Dependency Analysis to prepare for
promotion
Dependency Analysis enables the EME to
understand the required objects for a given
graph.

AB INITIO EME Developer View of
Promotion
Association of related objects is usually

automatic through dependency
analysis.
Can specify manually using:
air requires <rpath> []
e.g. AB_INCLUDE_FILES

AB INITIO EME
Tags and Configurations
Tag an object and its required files using

air tag create <tag> <object>
If you tag a project, all objects in the

project are tagged.

AB INITIO EME
air tag ls [-p | -e] <tag>
lists the objects designated by the tag.
-p lists the primary objects,

-e lists the required files as well.

AB INITIO EME
air tag import-configuration <config file>

<EME path>
imports a configuration file into the EME
air tag tag-configuration <tag> <config file>

creates a tag based on the configuration
file

AB INITIO EME
Promotion
Two approaches:
Check Out tagged objects to staging
sandbox, detach sandbox (if needed),
Check In to new EME
air save/air load commands for creating

portable formats

AB INITIO EME
Command: air save
The air save command exports selected

objects and relationships from the EME
into a flat file.
Dependency analysis information is saved
for graphs
Related objects and relationships
optionally can be saved
Project settings can optionally be saved

AB INITIO EME
Command: air load
The air load command imports data

saved to a flat file with air save.
Usage:
air load <filename>

AB INITIO EME
Validating Promotion
Validate promoted graphs by running

dependency analysis after loading into
the destination EME
Other graphs can be validated as well to

be sure that they havent been broken
by the new changes introduced.

AB INITIO EME Command: air project
analyze-dependencies
air project analyze-dependencies

runs dependency analysis on the server
without need for the GDE.

Ab Initio - Session6

Uploaded by

Copyright:

Available Formats

Ab Initio - Session6

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ab Initio - Session6

Uploaded by

Copyright:

Available Formats

AB INITIO EME

AB INITIO CONFIDENTIAL AND PROPRIETARY 2

All software and training material is:

AB INITIO CONFIDENTIAL AND PROPRIETARY 3

Using the EME...

AB INITIO CONFIDENTIAL AND PROPRIETARY 4

AB INITIO CONFIDENTIAL AND PROPRIETARY 5

AB INITIO CONFIDENTIAL AND PROPRIETARY 6

AB INITIO CONFIDENTIAL AND PROPRIETARY 7

Data about data.

Record formats, database ddl, field/column

Everything but the data itself.

AB INITIO CONFIDENTIAL AND PROPRIETARY 8

AB INITIO CONFIDENTIAL AND PROPRIETARY 9

AB INITIO CONFIDENTIAL AND PROPRIETARY 10

AB INITIO CONFIDENTIAL AND PROPRIETARY 11

Separate technical metadata

run - deployed scripts run

xfr - transforms xfr

AB INITIO CONFIDENTIAL AND PROPRIETARY 12

Divide technical metadata EDW

AB INITIO CONFIDENTIAL AND PROPRIETARY 13

Central Metadata Storage

AB INITIO CONFIDENTIAL AND PROPRIETARY 14

More about the EME...

from the developers

AB INITIO CONFIDENTIAL AND PROPRIETARY 15

AB INITIO CONFIDENTIAL AND PROPRIETARY 16

To use another projects parameters:

/Projects/stdenv Common Project /usr/admin/sand/stdenv

AB INITIO CONFIDENTIAL AND PROPRIETARY 17

Any project can be included

AB INITIO CONFIDENTIAL AND PROPRIETARY 18

A sandbox is a users personal

AB INITIO CONFIDENTIAL AND PROPRIETARY 19

Metadata separated Referring

dml - record formats dml $AI_DML

run - deployed scripts run $AI_RUN

xfr - transforms xfr $AI_XFR

AB INITIO CONFIDENTIAL AND PROPRIETARY 20

Sandbox is a file system copy of a EME

Project may have multiple copies in file

AB INITIO CONFIDENTIAL AND PROPRIETARY 22

Sandboxes are distinguished from

They may contain objects that are either

AB INITIO CONFIDENTIAL AND PROPRIETARY 23

Directory parameters establish map

Using the EME

from the GDE

AB INITIO CONFIDENTIAL AND PROPRIETARY 25

Developers - Source Code Control:

AB INITIO CONFIDENTIAL AND PROPRIETARY 26

Configuring the gde

AB INITIO CONFIDENTIAL AND PROPRIETARY 27

The Project>Settings dialog:

Click the Connect button.

AB INITIO CONFIDENTIAL AND PROPRIETARY 29

AB INITIO CONFIDENTIAL AND PROPRIETARY 30

Best Practice: Check Out