ODI 12c: Renaming a Project

Hello all,

Working as a consultant has its own challenges which makes me try things at home, which drives me to play more with Oracle and ODI and learn new things. Today a new challenge appeared in front of me. I am working in an upgrade project, where the upgrade from ODI 10g to ODI 12c is already completed and my task is to test the codes in ODI 12c to see if they are still good. (There are some problems but they will be topic of another post.)

So today, our client admitted that they do not have only ODI 10g to upgrade, they also have an ODI 12c repository other than the upgraded one. And they want to merge two 12c repositories together. But there are some problems.

  1. Projects have the same name
  2. Some variables have the same name
  3. Some custom KMs have the same name but different codes.

So we thought of some methods and their possible outcomes. One option was to change the Project Name (and Code off course) but we could not decide how will ODI react to this change. So I decided to give it a try in my training VM tonight.

What I was expecting?

  1. ODI will recognize and change codes of variables in mappings
  2. ODI will recognize and change codes of variables in packages (refresh, evaluate, set, declare variable components)
  3. ODI may recognize the variable usage in procedure options
  4. ODI will not recognize and change variables in procedures

Let’s see what happened together:

First I created a new project and I created a variable called MY_VAR and put some refresh code in it, which does not matter since I am not going to run any of the objects for this demo.

Project Creation

Project Creation

Second I created MY_PROC with one task TASK 1 and one option MY_OPT. Task has one line of code, which is the MY_VAR itself, just to see if it will change.

MY_VAR

MY_PROC

Then I created a mapping called MY_MAP, put two random datastores in it. Made a link between them and mapped a column with MY_VAR.

MY_MAP

MY_MAP

Then I created a package MY_PACK which includes a Refresh Variable, MY_PROC with MY_OPT set to MY_VAR, and finally MY_MAP.

MY_PACK

MY_PACK

Finally I proceed to change Project Name and Project Code. ODI warns user in this step, that errors in objects may occur especially with project variables.

Rename

Rename

After I accepted the changing the name and the code, let’s see what happened. Refresh Variable in the pack is changed as I expected. You can see its new path in the lower right corner of the image. (click on image to see it bigger)

Refresh MY_VAR

Refresh MY_VAR

Option MY_OPT of MY_PROC did not change.

MY_OPT

MY_OPT

Code in MY_PROC stays the same after change.

MY_PROC

MY_PROC

Column mapping in MY_MAP did not change also which is a disappointment for me.

MY_MAP

MY_MAP

After seeing the results let’s check how many guesses I’ve done right. I will copy/paste the list from above.

  1. ODI will recognize and change codes of variables in mappings DID NOT HAPPEN
  2. ODI will recognize and change codes of variables in packages (refresh, evaluate, set, declare variable components) HAPPENED
  3. ODI may recognize the variable usage in procedure options DID NOT HAPPEN
  4. ODI will not recognize and change variables in procedures HAPPENED

So I achieved a score of 2 out of 4 in my guesses. And unfortunately ODI achieved to detect only 1 of the four scenarios. So be cautious about changing the name and the code of a project in ODI, actually be careful about changing the code of any othe objects too. (such as models, contexts etc.)

This demo has been done using ODI 12.1.2.0.0 and results may vary depending on the version.

Version

Version

Thanks for reading and please do not hesitate to write your comments or ask questions in section below.

ODI 11g, ODI 12c : What’s an Agent?

What is an agent?

An agent is a java process which usually located on server and listens to a port for incoming requests. It runs the requested scenario, reverse engineers requested datastores etc.

When a job submitted through ODI Studio GUI or through startscen.sh agent gets scenario from work repository and topology definitions from master repository, combines and converts them into a runnable job, usually consisting more than one code block. Then it sends code blocks to destination environments, which may be DB servers, file servers, hadoop name node etc. Finally agent gets job statuses from these environments and writes into work repository tables for us to see from Operator tab of ODI Studio.

Agent diagram from Oracle A-Team Blog

Agent Types

Standalone Agent

It is the basic agent of ODI. It does not require an application server like JEE Agent. It is easy to configure/start/stop this agent from shell. Since today I’ve always used this agent and never tried other versions. A-Team article says this is the most light weight and low footprint choice.

JEE Agent

“Java Enterprise Edition” agent, which requires an application server, in most documentation you can see the name of WebLogic Server since it’s another Oracle product. (Some search results: depending on this CertMatrix of Oracle http://www.oracle.com/technetwork/middleware/data-integration/odi-11gr1certmatrix-ps6-1928216.xls ODI 11.1.1.7.0 only supports WLS and does not support Tomcat or other application servers. You may -or may not- configure them to run together but it is not supported.) This agent is first delivered with ODI 11g. And still exists in ODI 12c.

Some pros of JEE agent which are written in A-Team blog are:

  • High availability : Through Web Logic Server’s cluster architecture, even a node is down agents may run on other nodes.
  • Configurable connection pooling : Connection pool can be configured through WLS.
  • Monitoring : Oracle Enterprise Manager can monitor, configure, alert, manage ODI JEE Agents. But I believe, there is a plug-in to be installed to achieve this tasks from OEM.

Colocated Agent

The newest agent type which has arrived with ODI 12c. This type is like a combo of other two types. Agent is a standalone agent, but can be monitored and configured through WLS. Unfortunately it does not take advantage of connection pooling, high availability. Our agent will be in WLS domain, can be managed through WLS and that’s all. It is lighter than JEE Agent. In my opinion companies which prefers JEE Agent as production agents can choose colocated agent as their DEV/TEST agent.

Agent Types diagram from Gerard Nico's website

Agent Types diagram from Gerard Nico’s website

Where to locate an agent?

To decrease network I/O it is better to locate agent to target DB server. Since agent submits code to DB Engine, it is better for them to be on same machine. Don’t forget that ODI is an E-LT tool. Which means it will load data into target server, then it will transform your data. So most of the load will be on target server. Which also means most of the codes will be submitted to target server.

Also since an agent is a local java process, agent will write files to the machine, which it is set-up on. If you have a file server seperate than DB server, then it is better to have another agent on file server to handle read/write file processes. Or to mount file servers directory to DB server as a directory and setting up only one agent is another solution.

Also these solutions will prevent any firewall related problems.

 

Thanks for reading, don’t forget to share & comment.

 

 

Resources I’ve read before writing this post:
http://www.ateam-oracle.com/odi-agents-standalone-jee-and-colocated/
http://www.ateam-oracle.com/understanding-where-to-install-the-odi-standalone-agent/
http://odibeginnertutorials.blogspot.com.tr/2013/11/2-introduction-to-odi-agents-and.html
http://www.odigurus.com/2011/10/defining-odi-agents.html
http://gerardnico.com/wiki/dit/odi/agent

ODI 12c: Setting up Standalone Agent

Warning! This post is written by using ODI 12.1.2, in version ODI 12.1.3 there are changes in agent setup. A post about ODI 12.1.3 will be written soon.

Hello all,

In this post I am going to tell you how you can set up a standalone agent for ODI 12c. Oracle provided a new tool for configuring our ODI 12c setup elements, such as WLS, Console, Agent, RAC. But to use this configuration manager you will need to create some schemas for OPSS, AUDIT, AUDIT VIEWER and luckily we have RCU for this task, you may do it manual also. But I prefer automation and I will describe in this post how you do it with RCU.

Important! Notice that even if you have a database which can work with ODI 12c, it can be unsuitable with RCU. I faced this problem, I had Oracle 11.2.0.3 for my ODI repository but RCU needed a newer version, so I installed ODI 12c database.

Remember! The paths I will give in this post are absolute paths for my installations, your path may change depending on where you have installed ODI 12c.

Shortly we are going to do

  1. Create required repository and schemas by using RCU
  2. Create a domain to run agent by using Configuration Manager
  3. Define physical agent on ODI Studio
  4. Run agent
  5. Test it from ODI Studio

1. Creating necessary schemas

RCU

RCU

You will find RCU tool in /home/oracle/Oracle/Middleware/Oracle_Home/oracle_common/bin for Linux OS and C:\Oracle\Middleware\Oracle_Home\oracle_common\bin for Windows OS. File name is RCU.cmd or RCU.sh depending on OS. Start the RCU, select “Create Repository” and continue.

RCU

RCU

Fill all the spaces for the database you want to connect, be sure to enter full service name. In screenshot you will see it is filled as “orcl122” but full service name is “orcl122.localdomain”. I took the screenshot a bit early. Don’t let it trick you.

RCU

RCU

You will need to select schemas for RCU to create, Oracle Platform Security Services, Audit Services, Audit Services Viewer must be selected. When you check Audit Services, Audit Services Viewer and Audit Services Append will be checked automatically. Also in this step you can choose RCU to create Master and Work Repository for you. Since I have already created my repositories long ago I will skip it.

RCU

RCU

Set passwords for schemas that are going to be created. I will choose to set same password for every schema.

RCU

RCU

Set tablespace for schemas. If you leave it default, RCU will name them with a prefix which is set on third step. If tablespaces does not exist RCU will create them for you.

RCU

RCU

Now RCU will show you a summary of settings, check the list if it seem fine click on “Create”. It will take some time to create schemas, tablespaces, tables then it will show you a report. And you are ready to close RCU and continue with Configuration Manager.

2. Creating Domain for Agent with Configuration Manager

Conf Man

Conf Man

Start Configuration Manager, you can find it in /home/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/bin on Linux OS and in C:\Oracle\Middleware\Oracle_Home\oracle_common\common\bin in Windows OS. File name is config.sh or config.cmd depending on OS. Select “Create a new domain” and select where you want to put domain’s files. I will leave it as it is.

Conf Man

Conf Man

Next page is Template Selection page, we will need to check “Oracle Data Integrator – Standalone Collocated Agent” and when you check this option there will be two more options checked automatically.

Conf Man

Conf Man

Next step will want you to create a login for WLS. Password should be at least eight characters, including at least one letter and one number.

Conf Man

Conf Man

Select Domain Mode and JDK, I will leave them default.

Conf Man

Conf Man

Now we will select RCU Data option, since we created necessary schemas with RCU. Enter connection information for the database schema where RCU worked on before. Then click on “Get RCU Configuration” and wait for Configuration Manager to read from database. Then click “Next” when it is available.

Conf Man

Conf Man

Enter all passwords and schemas we have created before. Also you will need to fill database connection information again.

Conf Man

Conf Man

Configuration Manager will check for connections if they are working.

Conf Man

Conf Man

Now we need to select components which we are going to need, In this case we will only select “System Components” and “Deployments and Services” and continue.

Conf Man

Conf Man

Set name for the agent I will use OracleDemoAgent for this demo.

Config10

Conf Man

Next select the component from dropdown list, which we have created one step ago. Set listener information you can leave port as 21910 it is default for configuration manager. Fill in supervisor user information. And continue, for this demo we will leave settings default on other steps then click on “Create” on last step.

Conf Man

Conf Man

Creation will take sometime after that click on “Next” and “Finish” and we are almost ready to run the agent.

3. Defining Physical Agent in ODI Studio

Run ODI Studio, and login to repository, where you have Topology rights. We will need this rights to create agents.

Topology

Topology

Go to Topology tab, in Physical Architecture accordion, right click on “Agents” and click on “New Agent” a new window will appear on right panel.

Topology

Topology

Name the agent as the same name as we selected in Configuration Manager for the Agent component. When we try to run agent, it will search for its name in Master Repository so names should match exactly. Set the port as you set in Configuration Manager. Save the agent.

4. Start up the agent

Open a terminal/command line window and go to the domain folder that you have created. Under this folder you will find bin folder. For me this folders path is /home/oracle/Oracle/Middleware/Oracle_Home/user_projects/domains/demo_domain/bin in Linux OS or C:\Oracle\Middleware\Oracle_Home\user_projects\domains\demo_domain\bin in Windows OS. Run the command agent with NAME parameter as following, or as you can see in screenshot.  agent -NAME=OracleDemoAgent

Command Line

Command Line

Agent will start to rise up, it will connect to Master Repository, check for its name, read other configurations and start the service and listener. When agent is fully started, command line will seem like screenshot below.

Command Line

Command Line

5. Test the agent from ODI Studio

Now we have the agent up and running, also we have a definition for it in our Master Repository and a GUI in ODI Studio. It’s time to test if ODI Studio can connect to agent service. Go to Topology tab and open the agent we have created. There you will see a button written “Test” click on it. If everything is right until here you should see the screen below.

Test Agent

Test Agent

 

We have completed everything now. Agent is up and ready to run your jobs on databases for you.

Thanks for reading and please do not hesitate to write your comments or ask questions in section below.

 

 

 

ODI 12c: First Look and Repository Creation

Hello,

After a long pause on blog, here I am again. Oracle Data Integrator 12c is finally available for everyone to download. So in this post I will discuss about my first impressions and I will explain how to create repositories, both master and work. Actually it is pretty simple and almost same with 11g which I told in this post.

So first impressions, when you download ODI 12c through this page, you will get odi_121200.jar (numbers can differ with time since it’s version number) and some opatches bundled with it. Actually it is a bit disturbing for me to have a jar file which is 1.8GB. I’d like to have an exe for Windows.

Anyway I had some problems with running this jar also, first I tried it on my VM which has 32bit Windows 7 and got an error that states it could not reach jar file. So I moved to my physical machine that is 64bit Windows 7 and OUI could not recognize the platform and exited everytime, until I download and install Java 1.7. So after solving the problem with Java, I moved to my VM again to solve other problem where it came out that my path is problematic, since my user name is Canburak Tümer, space created a problem to reach file. I created another user without space that can run the installer.

Finally I could see the installer UI. It was a pretty straight forward installation, I just selected enterprise and went on. After installation, I ran the ODI Studio, it has a really clean and elegant splash screen and it asks to migrate any user settings from old installations. After splash screen, ODI workbench has been load:

ODI 12c Start Screen

ODI 12c Start Screen

Creating Master Repository

As I mentioned before, repository creations are almost same with ODI 11g, we will start by clicking File > New and we will see screen below:

Master - 1

Master – 1

Select “Master Repository Creation Wizard” in ODI tab and click “OK”.

Master - 2

Master – 2

You will see screen above, where we will enter database information, schema where we will create repository and DBA user to run some of the creation scripts.

Master - 3

Master – 3

Define and confirm password for SUPERVISOR user. DO NOT FORGET THIS PASSWORD UNLESS YOU HAVE ANOTHER USER WITH SUPERVISOR PRIVILEGES. For this reason many ODI developers/admins make this password “SUNOPSIS” as an old habit. I prefer to have it as “SUPERVISOR” in my VM and personal development environment.

Master - 4

Master – 4

Select password storage as you wish. Then click finish, it will run scripts now to create master repository, it took around 4 minutes in my VM, probably it will take around 2-3 minutes in your physical machines. Now it’s time to create a connection to master repository.

Connection

Connection

Click on “Connect to Repository” then click to green plus in the pop-up window, then fill required information in the form. Use SUPERVISOR as ODI user and DB user which you have created the master repository with. Make sure you have selected “Master repository only” radio button. Then click “OK”.

Wallet

Wallet

ODI 12c will ask you to if you want to keep passwords in a secure wallet with a master password. I do not have enough information about this wallet yet, but I will learn and write another post about it. I prefer the less secure way which does not include the wallet. Now we have master repository and connection to master repository. Now it’s time to create work repository.

Creating Work Repository

To create work repository, connect your master repository then go to Topology tab and expand Repositories menu.

Work - 1

Work – 1

Right click to “Work Repositories” and click to “New Work Repository” from menu.

Work - 2

Work – 2

Insert connection information of schema which you want to create Work Repository in. (I had a problem with this step, actually I wanted to use odiw_c user but ODI 12c keeps filling the form in upper case so it gives invalid credentials error.)

Work - 3

Work – 3

In final step, insert repository name and select repository type.

Work - 4

Work – 4

You can also define a password for repository, which is different from ODI user password or DB user password. This password is just to secure the repository connection. When you click “Finish” it will run scripts to create work repository and will ask you if you want to create a connection to work repository. It will create a connection without ODI user information. So you will need to edit connection to insert ODI user information.

After all these steps we have installed ODI 12c and setup both master and work repositories for our environment. And we have a final view as below :

We are ready to develop.

We are ready to develop.

Now, it’s time to create our topology connections, models, projects; import or reverse engineer data sources; develop mappings (new name for interfaces), packages and more.

Welcome to ODI 12c, keep following my blog for further posts and please do not hesitate to contact me through comment form below.