Oracle blog

In part 1, we discussed few things that you should take care before implementation of a ZDLRA. In this post, we will discuss few more things that you should review before or at the time of implementation: If you are getting two ZDLRAs (one each for primary and standby sites), there are two ways they can be deployed. One scenario is where all the primary databases (or the database that have no standby) backup to RA at the primary site and then the data is replicated from primary RA to RA at the standby site. This works well for the DBs that have no standby database. For the DBs where there is a standby database, there is a better architecture that can be deployed. In that scenario, primary databases backup to primary RA and the standby databases backup to standby RA. That saves you all the traffic over replication network. Oracle has published a whitepaper on how to do this configuration. Few of the instructions in this paper are a bit dated but it gives a good overall idea of how to do the implementation. Keep an eye on the features supported for different DB versions. An interesting one is that real-time redo shipping from standby databases is supported on 12c+ databases only. It is not supported for 11g. There could be other similar things. MOS note 1995866.1 has these details. Depending upon the ZDLRA software version being deployed, it may need a minimum version of EM and the ZDLRA plugin. MOS note 2542836.1 has these details. Make sure after discovering the the primary and standby databases in EM, their primary-standby relationship is reflected. Real-time redo sent to ZDLRA is compressed but the archive logs backup will be compressed only if you use compression in the RMAN command. It is always good to include backup archivelog command with daily incremental job to make sure that no archive log is missed. Many of the environments have separate networks for backup traffic. Make sure the backup traffic to ZDLRA uses DB server’s backup network. If that is not the case, you may need to add an explicit route on DB server for ZDLRA client/VIP/scan IPs. There are going to be different users that you will need to use: one OS user for deploying the EM agent, one DB user that will be used to run the backups. Depending upon your environment, it may oracle OS user, SYS DB user or could be some other named user created for this purpose. In next few posts, we will discuss some of the issues I have faced while doing ZDLRA implementation for some customers. ...

A quick note about an error I faced while running root.sh on an Exadata machine. The configuration tools failed with the following error: Error is PRVF-4657 : Name resolution setup check for "db-scan" (IP address: x.x.x.101) failed I did nslookup on the scan name and it all seemed good. So why the error ? After spending another 5 minutes, I looked at /etc/hosts and there was it. Someone had populated /etc/hosts of DB nodes with all the hostnames entries including the scan name. Something like: ...

Zero Data Loss Recovery Appliance (ZDLRA) is Oracle’s solution for database backups. It has many advantages over other backup solutions that are available in the market. This post has a brief introduction to ZDLRA and few links for further reading. This is a quick post about few of things that you should keep in mind if you are planning to get a ZDLRA (RA in short). Of course, there is a lot more that is needed while executing the whole plan, but these are some of the basics: ...

Exadata storage software version 20.1 introduces a new feature called “Secure Fabric” for KVM based multi cluster deployments (Exadata X8M). It enables network isolation between multiple tenants (i.e. KVM VMs based RAC clusters). This feature aligns with Infiniband Partitioning on OVM based systems. There are customers who in such scenarios want that VMs of one RAC shouldn’t be able to see traffic of the other RAC VMs. This feature achieves that. Similar to Pkeys in IB switches, here it uses a double VLAN tagging system where the first tag identiefies the network partition and the second tag is used to denote membership level of the VM. Exadata documention has more details. ...

There are two common scenarios when we may need this: An existing DB node has crashed and is unrecoverable (due to some failure and non-availability of any backups. Though some of the things may need to be done even if the backups were available). We have an existing Exadata rack that is virtualized. Now there is a new DB node and the existing clusters need to be extended to include the VMs on this new node. I recently faced the first scenario where a virtualized DB node crashed and wasn’t recoverable. A bare metal DB node restore is a relatively simple procedure where we just have to reimage the node, create the needed directories, users etc and add it to the RAC cluster. In case of virtualization, the creation of VMs is an additional step that needs to be done. That makes it slightly more complex. ...

I was patching an Exadata db node from 18.1.5.0.0.180506 to 19.3.2.0.0.191119. It had been more than an hour and dbnodeupdate.sh appeared to be stuck. Trying to ssh to the node was giving “connection refused” and the console had this output (some output removed for brevity): [ 458.006444] upgrade[8876]: [642/676] (72%) installing exadata-sun-computenode-19.3.2.0.0.191119-1... <> [ 459.991449] upgrade[8876]: Created symlink /etc/systemd/system/multi-user.target.wants/exadata-iscsi-reconcile.service, pointing to /etc/systemd/system/exadata-iscsi-reconcile.service. [ 460.011466] upgrade[8876]: Looking for unit files in (higher priority first): [ 460.021436] upgrade[8876]: /etc/systemd/system [ 460.028479] upgrade[8876]: /run/systemd/system [ 460.035431] upgrade[8876]: /usr/local/lib/systemd/system [ 460.042429] upgrade[8876]: /usr/lib/systemd/system [ 460.049457] upgrade[8876]: Looking for SysV init scripts in: [ 460.057474] upgrade[8876]: /etc/rc.d/init.d [ 460.064430] upgrade[8876]: Looking for SysV rcN.d links in: [ 460.071445] upgrade[8876]: /etc/rc.d [ 460.076454] upgrade[8876]: Looking for unit files in (higher priority first): [ 460.086461] upgrade[8876]: /etc/systemd/system [ 460.093435] upgrade[8876]: /run/systemd/system [ 460.100433] upgrade[8876]: /usr/local/lib/systemd/system [ 460.107474] upgrade[8876]: /usr/lib/systemd/system [ 460.114432] upgrade[8876]: Looking for SysV init scripts in: [ 460.122455] upgrade[8876]: /etc/rc.d/init.d [ 460.129458] upgrade[8876]: Looking for SysV rcN.d links in: [ 460.136468] upgrade[8876]: /etc/rc.d [ 460.141451] upgrade[8876]: Created symlink /etc/systemd/system/multi-user.target.wants/exadata-multipathmon.service, pointing to /etc/systemd/system/exadata-multipathmon.service. There was not much that I could do so just waited. Also created an SR with Oracle Support and they also suggested to wait. It started moving after some time and completed successfully. Finally when the node came up, i checked that there was an NFS mount entry in /etc/rc.local and that was what created the problem. For the second node, we commented this out and it was all smooth. Important to comment out all NFS entries during patching to avoid all such issues. I had commented the ones in /etc/fstab but the one in rc.local was an unexpected one. ...

I was installing Database Firewall version 12.2.0.11.0 on a Dell x86 machine (with 5 * 500 GB local HDDs configured in RAID 10) and it got successfully installed. Later on, I came to know that this version doesn’t support host monitor functionality on Windows hosts. The latest version that supports that is 12.2.0.10.0. So that was the time to download and install 12.2.0.10.0. The installation started fine but it failed with an error: ...

It was started by Tim Hall in 2016. This is a Thank you community post. There are so many experts posting on Oracle related forums, doing blog posts, sharing their scripts with everyone. All of you are doing a great job. I would like to mention three names especially: Tim Hall : Tim is a legend ! I don’t consider something a new feature until Tim writes about it :D Jonathan Lewis : I don’t think there is anyone on this planet who has even once worked on a performance problem and hasn’t gained something from the knowledge shared by him on forums or in one of the blog posts. ...

Use of Exadata storage cells seems to be a very poorly understood concept. A lot of people have confusions about how exactly ASM makes uses of disks from storage cells. Many folks assume there is some sort of RAID configured in the storage layer whereas there is nothing like that. I will try to explain some of the concepts in this post. Let’s take an example of an Exadata quarter rack that has 2 db and 3 storage nodes (node means a server here). Few things to note: ...

It is actually a dumb one. I was disabling triggers in a schema and ran this SQL to generate the disable statements. (Example from here) HR@test> select 'alter trigger '||trigger_name|| ' disable;' from user_triggers where table_name='PRODUCT'; 'ALTERTRIGGER'||TRIGGER_NAME||'DISABLE;' -------------------------------------------------------------------------------- alter trigger PRICE_HISTORY_TRIGGERv1 disable; HR@test> alter trigger PRICE_HISTORY_TRIGGERv1 disable; alter trigger PRICE_HISTORY_TRIGGERv1 disable * ERROR at line 1: ORA-04080: trigger 'PRICE_HISTORY_TRIGGERV1' does not exist HR@test> WTF ? It is there but the disable didn’t work. I was in hurry, tried to connect through SQL developer and disable and it worked ! Double WTF ! Then i spotted the problem. Someone created it with one letter in the name in small. So to make it work, we need to use double quotes. ...