Deep Copy of Audio CMSampleBuffer

Question

I am trying to create a copy of a CMSampleBuffer as returned by captureOutput in a AVCaptureAudioDataOutputSampleBufferDelegate.

The problem I am having is that my frames coming from delegate method captureOutput:didOutputSampleBuffer:fromConnection: being dropped after I retain them in CFArray for long time.

Obviously, I need to create deep copies of incoming buffers for further processing. I also know that CMSampleBufferCreateCopy only creates shallow copies.

There are few related questions were asked on SO:

But none of them helps me to use correctly CMSampleBufferCreate function with 12 parameters:

  CMSampleBufferRef copyBuffer;

  CMBlockBufferRef data = CMSampleBufferGetDataBuffer(sampleBuffer);
  CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer);
  CMItemCount itemCount = CMSampleBufferGetNumSamples(sampleBuffer);

  CMTime duration = CMSampleBufferGetDuration(sampleBuffer);
  CMTime presentationStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
  CMSampleTimingInfo timingInfo;
  timingInfo.duration = duration;
  timingInfo.presentationTimeStamp = presentationStamp;
  timingInfo.decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(sampleBuffer);


  size_t sampleSize = CMBlockBufferGetDataLength(data);
  CMBlockBufferRef sampleData;

  if (CMBlockBufferCopyDataBytes(data, 0, sampleSize, &sampleData) != kCMBlockBufferNoErr) {
    VLog(@"error during copying sample buffer");
  }

  // Here I tried data and sampleData CMBlockBuffer instance, but no success
  OSStatus status = CMSampleBufferCreate(kCFAllocatorDefault, data, isDataReady, nil, nil, formatDescription, itemCount, 1, &timingInfo, 1, &sampleSize, &copyBuffer);

  if (!self.sampleBufferArray)  {
    self.sampleBufferArray = CFArrayCreateMutable(NULL, 0, &kCFTypeArrayCallBacks);
    //EXC_BAD_ACCESS crash when trying to add sampleBuffer to the array
    CFArrayAppendValue(self.sampleBufferArray, copyBuffer);
  } else  {
    CFArrayAppendValue(self.sampleBufferArray, copyBuffer);
  }

How do you deep copy Audio CMSampleBuffer? Feel free to use any language (swift/objective-c) in your answers.

Is it obvious that you need a deep copy? What happens when you use `CMSampleBufferCreateCopy`? Would `CMSampleBufferCopySampleBufferForRange` give you a deep copy? Do you really need `CMSampleBuffer`s for further processing? If you're doing your own processing, length + pointer might be more convenient. — Rhythmic Fistman, Nov 02 '17 at 11:20
@RhythmicFistman yes, it is obvious that I need to deep copy, if I use `CMSampleBufferCreateCopy` and then retain the copied sample in `CFArray` more than 1s, `didOutputSampleBuffer` stops getting called. You can easily reproduce it with this [question](https://stackoverflow.com/questions/30850676/avcaptureoutput-didoutputsamplebuffer-stops-getting-called). I will check the behavior with `CMSampleBufferCopySampleBufferForRange` and will update you. — Neil Galiaskarov, Nov 02 '17 at 14:07
Ah, ok, that retaining buffers blocks your delegate callbacks is important information. Do you have a link to a runnable version of the above code? — Rhythmic Fistman, Nov 02 '17 at 14:14
There have been multiple questions on SO asking the same question, but none of them have a satisfying solution. I had the same problem in the past and couldn’t get it to work. In theory you could copy the underlying pixelbuffer, but it didn’t work when I last tried. Depending on your context I would suggest just writing to a file instead of buffering and then trimming the output video afterwards. At least that’s what I did. Be sure to post the solution here if you find a way to make this work. — , Oct 24 '17 at 11:25

score 15 · Accepted Answer · edited Dec 07 '18 at 05:52

Here is a working solution I finally implemented. I sent this snippet to Apple Developer Technical support and asked them to check if it is a correct way to copy incoming sample buffer. The basic idea is copy AudioBufferList and then create a CMSampleBuffer and set AudioBufferList to this sample.

AudioBufferList audioBufferList;
CMBlockBufferRef blockBuffer;
//Create an AudioBufferList containing the data from the CMSampleBuffer,
//and a CMBlockBuffer which references the data in that AudioBufferList.
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, NULL, &audioBufferList, sizeof(audioBufferList), NULL, NULL, 0, &blockBuffer);
NSUInteger size = sizeof(audioBufferList);
char buffer[size];

memcpy(buffer, &audioBufferList, size);
//This is the Audio data.
NSData *bufferData = [NSData dataWithBytes:buffer length:size];

const void *copyBufferData = [bufferData bytes];
copyBufferData = (char *)copyBufferData;

CMSampleBufferRef copyBuffer = NULL;
OSStatus status = -1;

/* Format Description */

AudioStreamBasicDescription audioFormat = *CMAudioFormatDescriptionGetStreamBasicDescription((CMAudioFormatDescriptionRef) CMSampleBufferGetFormatDescription(sampleBuffer));

CMFormatDescriptionRef format = NULL;
status = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, &audioFormat, 0, nil, 0, nil, nil, &format);

CMFormatDescriptionRef formatdes = NULL;
status = CMFormatDescriptionCreate(NULL, kCMMediaType_Audio, 'lpcm', NULL, &formatdes);
if (status != noErr)
{
  NSLog(@"Error in CMAudioFormatDescriptionCreator");
  CFRelease(blockBuffer);
  return;
}

/* Create sample Buffer */
CMItemCount framesCount = CMSampleBufferGetNumSamples(sampleBuffer);
CMSampleTimingInfo timing   = {.duration= CMTimeMake(1, 44100), .presentationTimeStamp= CMSampleBufferGetPresentationTimeStamp(sampleBuffer), .decodeTimeStamp= CMSampleBufferGetDecodeTimeStamp(sampleBuffer)};

status = CMSampleBufferCreate(kCFAllocatorDefault, nil , NO,nil,nil,format, framesCount, 1, &timing, 0, nil, &copyBuffer);

if( status != noErr) {
  NSLog(@"Error in CMSampleBufferCreate");
  CFRelease(blockBuffer);
  return;
}

/* Copy BufferList to Sample Buffer */
AudioBufferList receivedAudioBufferList;
memcpy(&receivedAudioBufferList, copyBufferData, sizeof(receivedAudioBufferList));

//Creates a CMBlockBuffer containing a copy of the data from the
//AudioBufferList.
status = CMSampleBufferSetDataBufferFromAudioBufferList(copyBuffer, kCFAllocatorDefault , kCFAllocatorDefault, 0, &receivedAudioBufferList);
if (status != noErr) {
  NSLog(@"Error in CMSampleBufferSetDataBufferFromAudioBufferList");
  CFRelease(blockBuffer);
  return;
}

Code-Level Support answer:

This code looks ok (though you’ll want to add some additional error checking). I've successfully tested it in an app that implements the AVCaptureAudioDataOutput delegate captureOutput:didOutputSampleBuffer:fromConnection: method to capture and record audio. The captured audio I'm getting when using this deep copy code appears to be the same as what I get when directly using the provided sample buffer (without the deep copy).

Apple Developer Technical Support

This is great, but I'm running into a similar issue with video samples (I need to retain the sample buffers). Do you have any tips on how to implement it for that? — Andy Hin, Jan 12 '18 at 18:03
Hi, I'm working on a project where i need to deep copy both video and audio. I've been able to deep copy video samplebuffer but cannot find a good solution for audio in Swift. Did you have the chance to convert your code in swift ? With your code, once you get the samplebuffer copy, how do you manage to write it to a file ? Thanks — user1523505, Sep 07 '18 at 08:33

score 0 · Answer 2 · answered Oct 22 '21 at 19:24

Couldn't find a decent answer doing this in Swift. Here's an extension:

extension CMSampleBuffer {
    func deepCopy() -> CMSampleBuffer? {
        guard let formatDesc = CMSampleBufferGetFormatDescription(self),
              let data = self.data else {
                  return nil
              }
        let nFrames = CMSampleBufferGetNumSamples(self)
        let pts = CMSampleBufferGetPresentationTimeStamp(self)
        let dataBuffer = data.withUnsafeBytes { (buffer) -> CMBlockBuffer? in
            var blockBuffer: CMBlockBuffer?
            let length: Int = data.count
            guard CMBlockBufferCreateWithMemoryBlock(
                allocator: kCFAllocatorDefault,
                memoryBlock: nil,
                blockLength: length,
                blockAllocator: nil,
                customBlockSource: nil,
                offsetToData: 0,
                dataLength: length,
                flags: 0,
                blockBufferOut: &blockBuffer) == noErr else {
                    print("Failed to create block")
                    return nil
                }
            guard CMBlockBufferReplaceDataBytes(
                with: buffer.baseAddress!,
                blockBuffer: blockBuffer!,
                offsetIntoDestination: 0,
                dataLength: length) == noErr else {
                    print("Failed to move bytes for block")
                    return nil
                }
            return blockBuffer
        }
        guard let dataBuffer = dataBuffer else {
            return nil
        }
        var newSampleBuffer: CMSampleBuffer?
        CMAudioSampleBufferCreateReadyWithPacketDescriptions(
            allocator: kCFAllocatorDefault,
            dataBuffer: dataBuffer,
            formatDescription: formatDesc,
            sampleCount: nFrames,
            presentationTimeStamp: pts,
            packetDescriptions: nil,
            sampleBufferOut: &newSampleBuffer
        )
        return newSampleBuffer
    }
}

Deep Copy of Audio CMSampleBuffer

2 Answers2

Linked