Wednesday, November 15, 2017

Unable to obtain current patch information due to error: 20001, ORA-20001: Latest xml inventory is not loaded into table

After upgrading single instance oracle restart setup to 12.2 from 11.2.0.4 following is observed in the alert log
Unable to obtain current patch information due to error: 20001, ORA-20001: Latest xml inventory is not loaded into table
ORA-06512: at "SYS.DBMS_QOPATCH", line 777
ORA-06512: at "SYS.DBMS_QOPATCH", line 864
ORA-06512: at "SYS.DBMS_QOPATCH", line 2222
ORA-06512: at "SYS.DBMS_QOPATCH", line 740
ORA-06512: at "SYS.DBMS_QOPATCH", line 2247
Checking in the $ORACLE_HOME/QOpatch/qopatch_log.log showed
KUP-05007:   Warning: Intra source concurrency disabled because the preprocessor option is being used.

Field Definitions for table OPATCH_XML_INV
  Record format DELIMITED BY NEWLINE
  Data in file has same endianness as the platform
  Reject rows with all null fields

  Fields in Data Source:

    XML_INVENTORY                   CHAR (100000000)
      Terminated by "UIJSVTBOEIZBEFFQBL"
      Trim whitespace same as SQL Loader
KUP-04095: preprocessor command /opt/app/oracle/product/12.2.0/dbhome_1/QOpatch/qopiprep.bat encountered error "/opt/app/oracle/product/12.2.0/dbhome_1/QOpatch/qopiprep.bat: 
line 55: /opt/app/oracle/product/12.2.0/dbhome_1/rdbms/log/stout_std11g2.txt: Permission denied
As per 1602089.1 this is due to running running the command with a different user who doesn't have permission on $ORACLE_HOME/QOpatch folder. But in this case it was not ture, the DB start was done by oracle user who has permissions on the QOPatch folder. Even then the same error was shown on alert log.



This is a role separate single instance setup and GI is owned by grid user. As mentioned in the password store post, in single instance setup with role separation has the problem of certain commands being run as grid user instead of oracle user. In this case the grid user doesn't have write permission on $ORACLE_HOME/rdbms/log folder which has 755 as the permission. That's the root cause of the issue. Change this log directory permission to 775 so that grid user is able to write to it. After this during the start up the full patch list is shown on the alert log.
2017-10-20T14:46:26.105603+01:00
===========================================================
Dumping current patch information
===========================================================
Patch Id: 26710464
Patch Description: Database Release Update : 12.2.0.1.171017 (26710464)
Patch Apply Time: 2017-10-20T13:05:21+01:00
Bugs Fixed: 14690846,17027695,17533661,19285025,19327292,19614243,20003668,
20324049,20620169,20736227,21159907,21186167,21981529,21985256,22072543,
22087683,22179537,22446455,22503297,22568728,22594071,22628825,22645009,
22654475,22729345,22898198,22950945,22981722,23026585,23035249,23055900,
23061453,23125560,23151677,23179662,23234232,23300142,23481673,23491861,
23499160,23521523,23527363,23548817,23581777,23599216,23665623,23730961,
23733981,23735292,23746128,23749454,24289874,24326846,24332831,24334708,
24336249,24341675,24368004,24376875,24376878,24385983,24421668,24425998,
24457597,24485161,24485174,24509056,24534401,24555417,24556967,24560906,
24573817,24578718,24578797,24589590,24609996,24624166,24642495,24655717,
24664211,24668398,24674955,24676172,24677696,24693290,24714096,24717183,
24735430,24737064,24737403,24744686,24792678,24796092,24811725,24812047,
24827228,24831514,24835919,24850622,24907917,24908321,24912588,24923215,
24929210,24938784,24942749,24960044,24968162,24976007,25029022,25034396,
...
...
The issue was only observed in single instance role separated setup. In role separated RAC configuration there wasn't any permission issues.

Useful metalink notes
Queryable Patch Inventory - Issues/Solutions for ORA-20001: Latest xml inventory is not loaded into table [ID 1602089.1]

Wednesday, November 1, 2017

Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 12.2.0.1

This post lists the steps for upgrading an Oracle restart environment (Single instance non-CDB on ASM) from 11.2.0.4 to 12.2.0.1. The 11.2.0.4 setup was on RHEL 6.4. The current versions of the GI are
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]

crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.4.0]
The Oracle restart environment used is a role separate setup where GI is installed as grid user while Oracle software is installed Oracle user. The GI and Oracle homes had the latest PSU applied (Database Patch Set Update : 11.2.0.4.171017 (26392168)). With this patch set applied orachk pre-upgrade did not list any patches to be applied before the upgrade. If the current patch set is different then additional patches may need to be applied before the upgrade. Refer 2180188.1 and earlier post of 11.2.0.4 to 12.2 RAC upgrade post for these patches.
Unlike the previous versions, 12.2 grid infrastructure requires unzipping it into a grid home. There's no GI home selection on the OUI. Create the new directory path for 12.2 GI and unzip the GI bundle.
mkdir -p /opt/app/oracle/product/12.2.0/grid
cp ~/linuxx64_12201_grid_home.zip /opt/app/oracle/product/12.2.0/grid
cd /opt/app/oracle/product/12.2.0/grid
unzip linuxx64_12201_grid_home.zip 
cluvfy from the 12.2 GI could be used to further check the system setup. This option is not for upgrade check but could be used to check readiness of oracle restart installation.
$ ./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/sbin,rhel6m1:/tmp ...PASSED
Verifying User Existence: grid ...
  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.20.51.0.2 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libgcc-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.4.7 (x86_64) ...PASSED
Verifying Package: sysstat-9.0.4 ...PASSED
Verifying Package: gcc-4.4.7 ...PASSED
Verifying Package: gcc-c++-4.4.7 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.81 ...PASSED
Verifying Package: glibc-2.12 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.12 (x86_64) ...PASSED
Verifying Package: libaio-0.3.107 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.107 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-5.43-1 ...PASSED
Verifying Package: net-tools-1.60-110 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.
The failure encountered is due to under size swap which could be ignored.
ASM will be upgraded as part of the upgrade process. Therefore stop the database before starting the GI upgrade. Upgrade process will prompt to stop the database during gridSetup run as well.
srvctl stop database -d std11g2
Start the GI upgrade by running the gridSetup.sh from the grid home.
$ ./gridSetup.sh
This will start the GI installer and select the upgrade GI option.
OS groups for authentication into ASM.
Oracle base location could be selected but not the GI Home location. This will be fixed, based on where the GI software is extracted.
Summary page and progress.
When prompted execute the rootupgrade.sh
# /opt/app/oracle/product/12.2.0/grid/rootupgrade.sh
Performing root user operation.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m1/crsconfig/roothas_2017-10-19_05-13-24PM.log
2017/10/19 17:13:26 CLSRSC-363: User ignored prerequisites during installation

ASM has been upgraded and started successfully.

Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel6m1 successfully pinned.
2017/10/19 17:17:38 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
CRS-4123: Oracle High Availability Services has been started.

2017/10/19 17:19:36 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p first'
2017/10/19 17:20:01 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p last'

rhel6m1     2017/10/19 17:20:04     /opt/app/oracle/product/12.2.0/grid/cdata/rhel6m1/backup_20171019_172004.olr     0

rhel6m1     2015/03/16 14:14:51     /opt/app/oracle/product/11.2.0/grid_4/cdata/rhel6m1/backup_20150316_141451.olr     -

rhel6m1     2015/03/11 17:57:18     /opt/app/oracle/product/11.2.0/grid_1/cdata/rhel6m1/backup_20150311_175718.olr     -
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/10/19 17:21:35 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
The GI versions after the upgrade
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]
crsctl query has softwareversion
Oracle High Availability Services version on the local node is [12.2.0.1.0]
The next step of the upgrade is to install 12.2 database software. 12.2 introduced a new OS group for RAC DB management. If this is used then add the user group to both oracle and grid. Not adding the group to grid user result in an issue which is mentioned in 12.1.0.2 to 12.2.1.0 single instance upgrade post.
# groupadd racdba
# usermod -g oinstall -G dba,oper,asmoper,asmdba,asmadmin,backupdba,dgdba,kmdba,racdba oracle
# id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),505(asmadmin),506(asmdba),507(asmoper),509(dgdba),510(kmdba),511(backupdba),512(racdba)
# usermod -g oinstall -G dba,asmoper,asmdba,asmadmin,racdba grid
id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),502(dba),505(asmadmin),506(asmdba),507(asmoper),512(racdba)
As GI home was install first some of the directories may not have write permission for oracle user. Change permission of those directories. In this case it was the 12.2.0 directory in the GI home path.
drwxr-xr-x. 3 grid oinstall 4096 Oct 19 16:44 12.2.0
$ chmod 775 12.2.0
drwxrwxr-x. 3 grid oinstall 4096 Oct 19 16:44 12.2.0
Install DB software selecting "software only" option.
RACDBA group selected for RAC administration.
Summary
Once the 12.2 DB software is installed next step is to upgrade the DB. 12.2 provides a preupgrade check jar file which need to be run using current oracle home (old version). Preupgrade.jar is available in 12.2_ORACLE_HOME/rdbms/admin
$ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
12.2.0.1.0

Upgrade-To version: 12.2.0.1.0

=======================================
Status of the database prior to upgrade
=======================================

      Database Name:  STD11G2
     Container Name:  Not Applicable in Pre-12.1 database
       Container ID:  Not Applicable in Pre-12.1 database
            Version:  11.2.0.4.0
         Compatible:  11.2.0.4.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  14
  Database log mode:  ARCHIVELOG
           Readonly:  FALSE
            Edition:  R1

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Enterprise Manager Repository   [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID


==============
BEFORE UPGRADE
==============

  Run /preupgrade_fixups.sql to complete all
  of the BEFORE UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
   + Adjust TABLESPACE SIZES as needed.
                                                Auto      12.2.0.1.0
     Tablespace                        Size     Extend    Min Size    Action
     ----------                     ----------  --------  ----------  ------

     SYSAUX                             600 MB  DISABLED      551 MB  None
     SYSTEM                             700 MB  ENABLED       806 MB  None
     TEMP                                20 MB  ENABLED       150 MB  None
     UNDOTBS1                           225 MB  ENABLED       400 MB  None

     Note that 12.2.0.1.0 minimum sizes are estimates.
     If you plan to upgrade multiple pluggable databases concurrently,
     then you must ensure that the UNDO tablespace size is equal to at least
     the number of pluggable databases that you upgrade concurrently,
     multiplied by that minimum.  Failing to allocate sufficient space can
     cause the upgrade to fail.

   + (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

     The database contains 2 objects in the recycle bin.

     The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
   + Remove the EM repository.

     - Copy the $ORACLE_HOME/rdbms/admin/emremove.sql script from the target
     12.2.0.1.0 ORACLE_HOME into the source 11.2.0.4.0 ORACLE_HOME.

     Step 1: If database control is configured, stop EM Database Control,
     using the following command

       $> emctl stop dbconsole

     Step 2: Connect to the database using the SYS account AS SYSDBA

       SET ECHO ON;
       SET SERVEROUTPUT ON;
      @emremove.sql

     Without the set echo and serveroutput commands, you will not be able to
     follow the progress of the script.

     The database has an Enterprise Manager Database Control repository.

     Starting with Oracle Database 12c, the local Enterprise Manager Database
     Control does not exist anymore. The repository will be removed from your
     database during the upgrade.  This step can be manually performed before
     the upgrade to reduce downtime.

   + Run 11.2.0.4.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
     objects.  You can view the individual invalid objects with

       SET SERVEROUTPUT ON;
       EXECUTE DBMS_PREUP.INVALID_OBJECTS;

     1 objects are INVALID.

     There should be no INVALID objects in SYS/SYSTEM or user schemas before
     database upgrade.

   + Backup the existing ACLs and their assignments for reference. Use the
     new DBMS_NETWORK_ACL_ADMIN interfaces and dictionary views to administer
     network privileges after upgrade.

     The database contains network ACLs with privileges that will be migrated
     to a new format in 12c.

     Network access control list (ACL) privileges in 11g will be migrated to
     a new format in 12c. As part of the migration, new
     DBMS_NETWORK_ACL_ADMIN interfaces and dictionary views are provided, and
     privileges in the existing ACLs will be converted to the new format with
     new ACL names. The old ACL names, DBMS_NETWORK_ACL_ADMIN interfaces and
     dictionary views may continue to be used but are deprecated and their
     use is discouraged. For further information, refer to My Oracle Support
     note number 2078710.1.

   + Review and remove any unnecessary EVENTS.

     The database contains events.

     There are events set that should be removed before upgrade, unless your
     application vendors and/or Oracle Support state differently.  Changes
     will need to be made in the spfile.

   + (AUTOFIXUP) Gather stale data dictionary statistics prior to database
     upgrade in off-peak time using:

      EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Dictionary statistics do not exist or are stale (not up-to-date).

     Dictionary statistics help the Oracle optimizer find efficient SQL
     execution plans and are essential for proper upgrade timing. Oracle
     recommends gathering dictionary statistics in the last 24 hours before
     database upgrade.

     For information on managing optimizer statistics, refer to the 11.2.0.4
     Oracle Database Performance Tuning Guide.

   + Directly grant ADMINISTER DATABASE TRIGGER privilege to the owner of the
     trigger or drop and re-create the trigger with a user that was granted
     directly with such. You can list those triggers using "SELECT OWNER,
     TRIGGER_NAME FROM DBA_TRIGGERS WHERE BASE_OBJECT_TYPE=''DATABASE'' AND
     OWNER NOT IN (SELECT GRANTEE FROM DBA_SYS_PRIVS WHERE
     PRIVILEGE=''ADMINISTER DATABASE TRIGGER'')"

     There is one or more database triggers whose owner does not have the
     right privilege on the database.

     The creation of database triggers must be done by users granted with
     ADMINISTER DATABASE TRIGGER privilege. Privilege must have been granted
     directly.

=============
AFTER UPGRADE
=============

  Run /postupgrade_fixups.sql to complete all
  of the AFTER UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
   + Upgrade the database time zone version using the DBMS_DST package.

     The database is using timezone datafile version 14 and the target
     12.2.0.1.0 database ships with timezone datafile version 26.

     Oracle recommends using the most recent timezone data.  For further
     information, refer to My Oracle Support Note 1585343.1.

   + (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Oracle recommends gathering dictionary statistics after upgrade.

     Dictionary statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans. After a
     database upgrade, statistics need to be re-gathered as there can now be
     tables that have significantly changed during the upgrade or new tables
     that do not have statistics gathered yet.

   + Gather statistics on fixed objects two weeks after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

     This recommendation is given for all preupgrade runs.

     Fixed object statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans.  Those
     statistics are specific to the Oracle Database release that generates
     them, and can be stale upon database upgrade.

Preupgrade generated files:
    /opt/app/oracle/cfgtoollogs/std11g2/preupgrade/preupgrade_fixups.sql
    /opt/app/oracle/cfgtoollogs/std11g2/preupgrade/postupgrade_fixups.sql
Before the upgrade the DB component status and versions, plus the timezone file version.
 COMP_ID    COMP_NAME                                VERSION         STATUS
---------- ---------------------------------------- --------------- --------
XDB        Oracle XML Database                      11.2.0.4.0      VALID
CONTEXT    Oracle Text                              11.2.0.4.0      VALID
OWM        Oracle Workspace Manager                 11.2.0.4.0      VALID
CATALOG    Oracle Database Catalog Views            11.2.0.4.0      VALID
CATPROC    Oracle Database Packages and Types       11.2.0.4.0      VALID


FILENAME                VERSION
-------------------- ----------
timezlrg_14.dat              14
Read the output from preupgrade.jar and carry out necessary pre-upgrade tasks. Some are included in the preupgrade_fixups.sql while others may required to be carried out manually. Once pre-upgrade tasks are completed run dbua from the 12.2 home to begin the upgrade.
Select the DB for upgrade. In this case type is indicated as SI (single instance)
Select the upgrade option. The timezone is upgraded along with the DB upgrade.
Select the listener. In this case the listener port is a non-default port.
Upgrade summary and progress
Post upgrade summary
Run the post upgrade script
@postupgrade_fixups.sql
DB components versions and status after the upgrade
COMP_ID    COMP_NAME                                VERSION    STATUS
---------- ---------------------------------------- ---------- --------
CATALOG    Oracle Database Catalog Views            12.2.0.1.0 VALID
CATPROC    Oracle Database Packages and Types       12.2.0.1.0 VALID
OWM        Oracle Workspace Manager                 12.2.0.1.0 VALID
CONTEXT    Oracle Text                              12.2.0.1.0 VALID
XDB        Oracle XML Database                      12.2.0.1.0 VALID
Timezone file version after upgrade
FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_26.dat              26          0
As mentioned in 11.2.0.4 to 12.2.0.1 RAC upgrade post, during the upgrade new user accounts are added to DB with default passwords. Change the default password of these user accounts.

Check the registry history view for the upgrade entry.
ACTION_TIME                    ACTION               NAMESPACE   VERSION            ID COMMENTS
------------------------------ -------------------- ----------- ---------- ---------- ----------------------------------
11-MAR-15 06.48.08.733830 PM   APPLY                SERVER      11.2.0.3            0 Patchset 11.2.0.2.0
12-MAR-15 11.54.22.212281 AM   APPLY                SERVER      11.2.0.3           13 PSU 11.2.0.3.13
16-MAR-15 03.09.19.557408 PM   VIEW INVALIDATE                                8289601 view invalidation
16-MAR-15 03.09.19.634383 PM   UPGRADE              SERVER      11.2.0.4.0            Upgraded from 11.2.0.3.0
16-MAR-15 03.09.43.891843 PM   APPLY                SERVER      11.2.0.4            0 Patchset 11.2.0.2.0
10-AUG-15 04.28.56.297048 PM   APPLY                SERVER      11.2.0.4            7 PSU 11.2.0.4.7
19-OCT-17 04.22.55.222389 PM   APPLY                SERVER      11.2.0.4       171017 PSU 11.2.0.4.171017
                               BOOTSTRAP            DATAPATCH   12.2.0.1              RDBMS_12.2.0.1.0_LINUX.X64_170125
19-OCT-17 06.40.33.227204 PM   VIEW INVALIDATE                                8289601 view invalidation
19-OCT-17 06.42.41.868410 PM   UPGRADE              SERVER      12.2.0.1.0            Upgraded from 11.2.0.4.0
Upgrade the database and ASM compatible parameters
alter system set compatible='12.2.0.1.0' scope=spfile sid='*';

srvctl stop database -d std11g2

alter diskgroup flash SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup DATA  SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup flash set attribute 'compatible.rdbms'='12.2.0.1.0';
alter diskgroup data set attribute 'compatible.rdbms'='12.2.0.1.0';

GROUP_NUMBER NAME                 VALUE
------------ -------------------- ---------------
           1 compatible.asm       12.2.0.1.0
           1 compatible.rdbms     12.2.0.1.0
           2 compatible.asm       12.2.0.1.0
           2 compatible.rdbms     12.2.0.1.0

srvctl start database -d std11g2
Finally run orachk and cluvfy for check post upgrade state
./orachk -u -o post

cluvfy stage -post hacfg

Verifying Oracle Restart Integrity ...PASSED
Verifying OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.
Useful metalink notes
ACFS Drivers Install reports CLSRSC-400: A system reboot is required to continue installing [ID 2025056.1]
Patches to apply before upgrading Oracle GI and DB to 12.2.0.1 [ID 2180188.1]
12.2 Grid Infrastructure Installation: What's New [ID 2024946.1]
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]

Related Posts
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 12.1.0.2 to 12.2.0.1
Upgrading Single Instance on ASM from 11.2.0.3 to 11.2.0.4
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2

Wednesday, October 25, 2017

Downgrade Grid Infrastructure from 12.2.0.1 to 12.1.0.2

This post list steps for downgrading grid infrastructure from 12.2.0.1 to 12.1.0.2. The grid infrastructure upgrade has been successful on all nodes. If upgrade to 12.2 had failed on any of the nodes then follow oracle documentation for additional steps/options required for such cases. Only the GI was upgraded to 12.2 and the database remained on 12.1.0.2. The current version of the cluster and nodes is as below
$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]

$ crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel12c1] is [12.2.0.1.0]
Oracle Clusterware version on node [rhel12c2] is [12.2.0.1.0]
1. As the first step for downgrade, remove the MGMTDB.
[grid@rhel12c1 ~]$ dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
52% complete
Deleting instance and datafiles
76% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb.log" for further details.
2. Run rootcrs.sh with downgrade option from any of the nodes
[root@rhel12c1 grid]# ./crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel12c1/crsconfig/crsdowngrade_rhel12c1_2017-09-25_04-26-42PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c1'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/09/25 16:28:17 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:28:43 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:28:43 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:29:21 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:29:21 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
3. Run the downgrade from the last node.
[root@rhel12c2 grid]# ./crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel12c2/crsconfig/crsdowngrade_rhel12c2_2017-09-25_04-30-45PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSFS.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.DATA.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.FRA.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.GIMR.dg' is already running on 'rhel12c2'
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.std12c1.db' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2676: Start of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2674: Start of 'ora.drivers.acfs' on 'rhel12c2' failed
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
2017/09/25 16:39:54 CLSRSC-338: Successfully downgraded OCR to version 12.1.0.2.0
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
2017/09/25 16:40:24 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:40:50 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:40:50 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:41:25 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:41:25 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2017/09/25 16:41:25 CLSRSC-640: To complete the downgrade operation, ensure that the node inventory on all nodes points to the configured Grid Infrastructure home '/opt/app/12.1.0/grid2'.
2017/09/25 16:41:27 CLSRSC-592: Run 'crsctl start crs' from home /opt/app/12.1.0/grid2 on each node to complete downgrade.



4. Upgrade the oracle inventory by removing the CRS association from 12.2 GI home and adding it to 12.1 home.
[grid@rhel12c1 bin]$ pwd
/opt/app/12.2.0/grid/oui/bin

./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.2.0/grid "CLUSTER_NODES=rhel12c1,rhel12c2" -doNotUpdateNodeList 
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5119 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

[grid@rhel12c1 bin]$ pwd
/opt/app/12.1.0/grid2/oui/bin

./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES=rhel12c1,rhel12c2"

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5119 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
Verify the update on inventory.xml
<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel12c1"/>
      <NODE NAME="rhel12c2"/>
   </NODE_LIST>
</HOME>

<HOME NAME="OraGI12Home2" LOC="/opt/app/12.2.0/grid" TYPE="O" IDX="3"/>
5. If the /etc/oratab is missing any ASM references with downgraded 12.1 GI home, add the entries manually.

6. Before re-creatign the 12.1 MGMTDB check if there's any references to mgmtdb (using crsctl stat res -t). If there are any references then recreating the MGMTDB will fail with
PRCD-1000 : Database _mgmtdb already exists
PRCR-1086 : resource ora.mgmtdb is already registered
mgmtdb could be removed with
srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y
7. Once all residual information on mgmtdb is removed create the MGMTDB root container.
[grid@rhel12c1 ~]$ $GI_HOME/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName +GIMR -datafileJarLocation $GI_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
 Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.
8. Create the PDB inside the MGMTDB root. The PDB should be named same as the cluster name. Cluster name could be found out with
$ cemutlo -n
prod-cluster
If cluster name has a dash (-) then when specify the PDB name replace the dash (-) with an underscore (_). Without this change the creation of the PDB will fail. More on this was mentioned on an earlier post.
[grid@rhel12c1 ~]$ $GI_HOME/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName prod_cluster -createPDBFrom RMANBACKUP -PDBBackUpfile  $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile  $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck

4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/prod_cluster/_mgmtdb0.log" for further details.
9. Finally run mgmtca and verify mgmtdb resoruce is available
run mgmtca

ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
10. As mentioned in upgarde to 12.2 the SCAN name changes if GNS is used. This cluster uses GNS. Check if the SCAN name has been reverted back to 12.1 format.
srvctl config scan
SCAN name: prod-cluster-scan.rac.domain.net, Network: 1
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
SCAN 0 IPv4 VIP: -/scan1-vip/192.168.0.92
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 1 IPv4 VIP: -/scan2-vip/192.168.0.89
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 2 IPv4 VIP: -/scan3-vip/192.168.0.90
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
This concludes the downgrade to 12.1.0.2 from 12.2.0.1

Related posts
Downgrade Grid Infrastructure from 12.2.0.1 to 11.2.0.4
Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4
Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7

Useful metalink notes
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]