ASMCMD> cp annnf0_ARCHIVE_LOG_0.382.923545643 /mnt/dbbackup copying +flash/livedb/backupset/2016_09_26/annnf0_ARCHIVE_LOG_0.382.923545643 -> /mnt/dbbackup/annnf0_ARCHIVE_LOG_0.382.923545643 ASMCMD-08016: copy source->'+flash/livedb/backupset/2016_09_26/annnf0_ARCHIVE_LOG_0.382.923545643' and target->'/mnt/dbbackup/annnf0_ARCHIVE_LOG_0.382.923545643' failed ORA-19510: failed to set size of 173851 blocks for file "/mnt/dbbackup/annnf0_ARCHIVE_LOG_0.382.923545643" (block size=512) ORA-27045: unable to close the file Linux-x86_64 Error: 5: Input/output error Additional information: 4 ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 258 ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)However if the copied location is examined the file is there with same size as source. However integrity of the file cannot be guaranteed as cp command has failed. The error message suggests "unable to close the file" so it could be that file is copied but file descriptor could not be closed. Copying to other file systems (ext3) worked fine.
ASMCMD> cp annnf0_ARCHIVE_LOG_0.382.923545643 /home/oracle copying +flash/livedb/backupset/2016_09_26/annnf0_ARCHIVE_LOG_0.382.923545643 -> /home/oracle/annnf0_ARCHIVE_LOG_0.382.923545643So issue was assumed not to be with asmcmd but the file system.
Trying to create a backup on the CIFS location also failed
RMAN> backup current controlfile format '/mnt/dbbackup/controlc.tl'; Starting backup at 28-Sep-2016 09:14:12 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=1029 instance=livedb2 device type=DISK channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set including current control file in backup set channel ORA_DISK_1: starting piece 1 at 28-Sep-2016 09:14:15 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03009: failure of backup command on ORA_DISK_1 channel at 09/28/2016 09:14:16 ORA-19504: failed to create file "/mnt/dbbackup/controlc.tl" ORA-27040: file create error, unable to create file Linux-x86_64 Error: 22: Invalid argument
Looking in MOS several notes were found which suggest limitations of using certain Oracle utilities with CIFS. For backups it was suggested to change the filesystemio_options to asynch or failing that to mount the CIFS location with file type cifs and oracle user's user id , group id
mount -t cifs -o username=oracle,uid=oracle,gid=oinstall //cfis location /mount pointSecond solution had already been implemented and asmcmd cp and rman backup still failed. However after changing filesystemio_options to asynch, rman backups succeded but asmcmd cp continue to fail. It seems asmcmd is another utility that cannot be used with CIFS (disclaimer: could not find any official oracle document to verify this though).
Beside asmcmd , datapump is also not supported on CIFS (793805.1). CIFS could be used for RMAN backups but it is not a certified file system. Which means if there's any issue when using RMAN with CIFS there won't be any oracle support to resolve the issue (444809.1).
Useful Metalink Notes
Data Pump Fails With Errors ORA-39097 ORA-39065 ORA-31643 ORA-19510 Using CIFS [793805.1]
Is use of CIFS Protocol for RMAN backups supported? [444809.1]
'cp' Fails with 'cp: cannot create symbolic link' When Copying Files to CIFS Filesystem [728223.1]
ORA-19504 ORA-27040 During Rman Backup To A Mounted Windows Directory In Linux [464267.1]
Which are the mount settings for SAMBA? [557057.1]
Product Support Document for Common Internet File System (CIFS)[1398067.1]
How to mount Windows share on Red Hat Enterprise Linux system using CIFS?