Android 4.0 Ice Cream Sandwich Media Framework (4)

Continued (4)

Start NuPlayer tracing

---===NuPlayer===---



Recall in Android 4.0 Ice Cream Sandwich Media Framework (1), if the data source URL contains m3u8 (ex: http://192.168.1.3/playlist.m3u8), or start with rtsp protocol, then NU_PLAYER will be selected by MediaPlayerService.

Here, the NuPlayer tracing will focus on HTTP Live Streaming, which means the m3u8 file decode.

Before start the code tracing, we first take a look about the m3u8 file.
(the URL to set for MediaPlayer in APK is like:http://192.168.1.3/playlist.m3u8 )

==playlist.m3u8 =================================

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID:1,bandwidth=600000
http://192.168.1.22/playhigh.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID:1,bandwidth=200000
http://192.168.1.21/playlow.m3u8

==playhigh.m3u8=================================

#EXTM3U
#EXT-X-TARGETDURATION:5
#EXTINF:5,
http://192.168.1.22/high/sample-no-1.ts
#EXTINF:5,
http://192.168.1.22/high/sample-no-2.ts
#EXTINF:5,
http://192.168.1.22/high/sample-no-3.ts
#EXT-X-ENDLIST

==playlow.m3u8=================================

#EXTM3U
#EXT-X-TARGETDURATION:5
#EXTINF:5,
http://192.168.1.21/low/sample-no-1.ts
#EXTINF:5,
http://192.168.1.21/low/sample-no-2.ts
#EXTINF:5,
http://192.168.1.21/low/sample-no-3.ts


#EXT-X-ENDLIST
=============================================

All the tags like "#EXTM3U" and others are defined in Apple's HTTP Live Streaming protocol

see the following website for more information
http://tools.ietf.org/html/draft-pantos-http-live-streaming-07

The introduction of HTTP Live Streaming

see the following website for more information
http://developer.apple.com/library/ios/#documentation/NetworkingInternet/Conceptual/StreamingMediaGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40008332-CH1-DontLinkElementID_39

And, the supported version of HTTP Live Streaming in Android :
http://developer.android.com/guide/appendix/media-formats.html


==================================================================
Let's get it started.

In APK, we use the same process to start media playback

            mMediaPlayer = new MediaPlayer();
            mMediaPlayer.setDataSource(path);
            mMediaPlayer.prepare();
            mMediaPlayer.start();



In Media Framework  :android_src\framework\base\media\libmediaplayerservice\MediaPlayerService.cpp

static sp createPlayer(player_type playerType, void* cookie,
        notify_callback_f notifyFunc)
{
    sp p;
    switch (playerType) {
        case SONIVOX_PLAYER:
            LOGD(" create MidiFile");
            p = new MidiFile();
            break;
        case STAGEFRIGHT_PLAYER:
            LOGD(" create StagefrightPlayer");
            p = new StagefrightPlayer;
            break;
        case NU_PLAYER:
            LOGD(" create NuPlayer");
            p = new NuPlayerDriver;
            break;
        case TEST_PLAYER:
            LOGD("Create Test Player stub");
            p = new TestPlayerStub();
            break;
        default:
            LOGD("Unknown player type: %d", playerType);
            return NULL;
    }
}

Just mentioned before, if the URL contains m3u8, the NU_PLAYER will be selected. Therefore, here NuPlayerDriver will be newed.

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\NuPlayerDriver.cpp

NuPlayerDriver::NuPlayerDriver()
    : mDurationUs(-1),
      mPositionUs(-1),
      mNumFramesTotal(0),
      mNumFramesDropped(0),
      mState(UNINITIALIZED),
      mLooper(new ALooper),{
    mLooper->setName("NuPlayerDriver Looper");


    mLooper->start();


    mPlayer = new NuPlayer;
    mLooper->registerHandler(mPlayer);


    mPlayer->setDriver(this);
}

NuPlayerDriver's constructor initialize the Looper/Message/Handler mechanism for NuPlayer.

The role of NuPlayerDriver is like StagefrightPlayer, the interface with MediaPlayer.

mPlayer->setDriver(this) let NuPlayer has the ability to send messages back.(media durations, number of frames, etc).

Note here mState is set as UNINITIALIZED.

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\NuPlayer.cpp

NuPlayer::NuPlayer()
    : mVideoIsAVC(false),
      mAudioEOS(false),
      mVideoEOS(false),
      mVideoLateByUs(0ll),
      mNumFramesTotal(0ll),
      mNumFramesDropped(0ll) {
}

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\NuPlayerDriver.cpp

status_t NuPlayerDriver::setDataSource(
        const char *url, const KeyedVector *headers) {


    CHECK_EQ((int)mState, (int)UNINITIALIZED);


    mPlayer->setDataSource(url, headers);


    mState = STOPPED;


}

Note here mState is set as  STOPPED .

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\NuPlayer.cpp

void NuPlayer::setDataSource(
        const char *url, const KeyedVector *headers) {
    sp msg = new AMessage(kWhatSetDataSource, id());


    if (!strncasecmp(url, "rtsp://", 7)) {
        msg->setObject(
                "source", new RTSPSource(url, headers, mUIDValid, mUID));
    } else {
        msg->setObject(
                "source", new HTTPLiveSource(url, headers, mUIDValid, mUID));
    }


    msg->post();
}

If url start with "rtsp://", then RTSPSource will be instantiated, else (for m3u8)HTTPLiveSource will be instantiated.

Both of them are used to handle the incoming media stream.

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\HTTPLiveSource.cpp

NuPlayer::HTTPLiveSource::HTTPLiveSource(
        const char *url,
        const KeyedVector *headers,
        bool uidValid, uid_t uid)
    : mURL(url),
      mFlags(0),
      mFinalResult(OK),
      mOffset(0) {
}

Then a message  kWhatSetDataSource posted. Note that the message contains the instance of RTSPSource or HTTPLiveSource.

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\NuPlayer.cpp

void NuPlayer::onMessageReceived(const sp &msg) {
    switch (msg->what()) {
        case kWhatSetDataSource:
        {
            LOGV("kWhatSetDataSource");


            CHECK(mSource == NULL);


            sp obj;
            CHECK(msg->findObject("source", &obj));


            mSource = static_cast(obj.get());
            break;
        }
    }
}

Here, mSource refer to HTTPLiveSource or RTSPSource

Now we finish the mMediaPlayer.setDataSource(path);

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\MediaPlayerService.cpp

status_t MediaPlayerService::Client::prepareAsync()
{
    LOGV("[%d] prepareAsync", mConnId);
    sp p = getPlayer();
    if (p == 0) return UNKNOWN_ERROR;
    status_t ret = p->prepareAsync();
    return ret;
}

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\NuPlayerDriver.cpp

status_t NuPlayerDriver::prepareAsync() {
    notifyListener(MEDIA_PREPARED);


    return OK;
}

The prepareAsync() here do nothing but notify back MEDIA_PREPARED

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\MediaPlayerService.cpp

status_t MediaPlayerService::Client::start()
{
    LOGV("[%d] start", mConnId);
    sp p = getPlayer();


    return p->start();
}

Note that mState of NuPlayerDriver is set as STOPPED now.

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\NuPlayerDriver.cpp

status_t NuPlayerDriver::start() {
    switch (mState) {
        case STOPPED:
        {
            mAtEOS = false;
            mPlayer->start();


            break;
        }


    mState = PLAYING;
    return OK;
}

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\NuPlayer.cpp

void NuPlayer::start() {
    (new AMessage(kWhatStart, id()))->post();
}

NuPlayer use message to post kWhatStart.

void NuPlayer::onMessageReceived(const sp &msg) {
    switch (msg->what()) {
        case kWhatStart:
        {
            LOGV("kWhatStart");


            mVideoIsAVC = false;
            mAudioEOS = false;
            mVideoEOS = false;
            mSkipRenderingAudioUntilMediaTimeUs = -1;
            mSkipRenderingVideoUntilMediaTimeUs = -1;
            mVideoLateByUs = 0;
            mNumFramesTotal = 0;
            mNumFramesDropped = 0;


            mSource->start();


            mRenderer = new Renderer(
                    mAudioSink,
                    new AMessage(kWhatRendererNotify, id()));


            looper()->registerHandler(mRenderer);


            postScanSources();
            break;
        }
    }
}

In Media Framework  :android_src\framework\base\media\libmediaplayerservice\nuplayer\HTTPLiveSource.cpp

void NuPlayer::HTTPLiveSource::start() {
    mLiveLooper = new ALooper;
    mLiveLooper->setName("http live");
    mLiveLooper->start();


    mLiveSession = new LiveSession(
            (mFlags & kFlagIncognito) ? LiveSession::kFlagIncognito : 0,
            mUIDValid, mUID);


    mLiveLooper->registerHandler(mLiveSession);


    mLiveSession->connect(
            mURL.c_str(), mExtraHeaders.isEmpty() ? NULL : &mExtraHeaders);


    mTSParser = new ATSParser;
}

HTTPLiveSource create it own Looper for Message/Handler mechanism, and instantiate the LiveSession

In Media Framework  :android_src\framework\base\media\libstagefright\httplive\LiveSession.cpp

LiveSession::LiveSession(uint32_t flags, bool uidValid, uid_t uid)
    : mDataSource(new LiveDataSource),
      mHTTPDataSource(
              HTTPBase::Create(
                  (mFlags & kFlagIncognito)
                    ? HTTPBase::kFlagIncognito
                    : 0)),
      mPrevBandwidthIndex(-1),
      mLastPlaylistFetchTimeUs(-1),
      mSeqNumber(-1),
      mSeekTimeUs(-1),
      mNumRetries(0),
      mDurationUs(-1),
      mSeekDone(false),
      mDisconnectPending(false),
      mMonitorQueueGeneration(0),
      mRefreshState(INITIAL_MINIMUM_RELOAD_DELAY) {
}

mDataSource is an instance of LiveDataSource, which acts as the buffer queue manager. we'll see how it work later.

In Media Framework  :android_src\framework\base\media\libstagefright\httplive\LiveDataSource.cpp

LiveDataSource::LiveDataSource()
    : mOffset(0),
      mFinalResult(OK),
      mBackupFile(NULL) {
}

In Media Framework  :android_src\framework\base\media\libstagefright\HTTPBase.cpp

sp HTTPBase::Create(uint32_t flags) {
        return new ChromiumHTTPDataSource(flags);
}

So, the mHTTPDataSource in LiveSession is an instance of ChromiumHTTPDataSource.

ChromiumHTTPDataSource will help LiveSession to read data from internet.

In Media Framework  :android_src\framework\base\media\libstagefright\chromium_http\ChromiumHTTPDataSource.cpp

ChromiumHTTPDataSource::ChromiumHTTPDataSource(uint32_t flags)
    : mFlags(flags),
      mState(DISCONNECTED),
      mDelegate(new SfDelegate),
      mCurrentOffset(0),
      mIOResult(OK),
      mContentSize(-1),
      mDecryptHandle(NULL),
      mDrmManagerClient(NULL) {
      mDelegate->setOwner(this);
LOG_PRI(ANDROID_LOG_VERBOSE, LOG_TAG,"ChromiumHTTPDataSource ctor");
}

Now back to NuPlayer::HTTPLiveSource::start() and do mLiveSession->connect()

In Media Framework  :android_src\framework\base\media\libstagefright\httplive\LiveSession.cpp

void LiveSession::connect(
        const char *url, const KeyedVector *headers) {
    sp msg = new AMessage(kWhatConnect, id());
    msg->setString("url", url);


    if (headers != NULL) {
        msg->setPointer(
                "headers",
                new KeyedVector(*headers));
    }


    msg->post();
}

We pause here and take a look of new ATSParser, used for future parsing.

In Media Framework  :android_src\framework\base\media\libstagefright\mpegts\ATSParser.cpp

ATSParser::ATSParser(uint32_t flags)
    : mFlags(flags) {
}

ATSParser 's constructor doesn't do anything special, so we continue to see how to handle the message kWhatConnect posted in LiveSession::connect()

void LiveSession::onMessageReceived(const sp &msg) {
    switch (msg->what()) {
        case kWhatConnect:
            onConnect(msg);
            break;
    }
}

void LiveSession::onConnect(const sp &msg) {
    CHECK(msg->findString("url", &url));


    bool dummy;
    sp playlist = fetchPlaylist(url.c_str(), &dummy);


    if (playlist->isVariantPlaylist()) {
        for (size_t i = 0; i < playlist->size(); ++i) {
            BandwidthItem item;


            sp meta;
            playlist->itemAt(i, &item.mURI, &meta);


            unsigned long bandwidth;
            CHECK(meta->findInt32("bandwidth", (int32_t *)&item.mBandwidth));
            LOGD("Check Bandwidth : %lu",item.mBandwidth);
            mBandwidthItems.push(item);
        }


        CHECK_GT(mBandwidthItems.size(), 0u);


        mBandwidthItems.sort(SortByBandwidth);
    }else{
        LOGD("Not VariantPlayList");
    }


    postMonitorQueue();
}


sp LiveSession::fetchPlaylist(const char *url, bool *unchanged) {
    *unchanged = false;
    status_t err = fetchFile(url, &buffer);
    sp playlist =
        new M3UParser(url, buffer->data(), buffer->size());


    return playlist;
}

fetchPlaylist here will read the data of m3u8 file from internet, and instantiate M3UParser for parsing the content of m3u8 file

playlist->isVariantPlaylist() judge if the playlist contains the tag "#EXT-X-STREAM-INF"

If contains, the playlist has the bandwidth information in the playlist.

In Media Framework  :android_src\framework\base\media\libstagefright\httplive\M3UParser.cpp

M3UParser::M3UParser(
        const char *baseURI, const void *data, size_t size)
    : mInitCheck(NO_INIT),
      mBaseURI(baseURI),
      mIsExtM3U(false),
      mIsVariantPlaylist(false),
      mIsComplete(false) {
    mInitCheck = parse(data, size);
}

The parse() in M3UParser analyzes the tags in m3u8 file.

In Media Framework  :android_src\framework\base\media\libstagefright\httplive\LiveSession.cpp

void LiveSession::postMonitorQueue(int64_t delayUs) {
    sp msg = new AMessage(kWhatMonitorQueue, id());
    msg->setInt32("generation", ++mMonitorQueueGeneration);
    msg->post(delayUs);
}

void LiveSession::onMessageReceived(const sp &msg) {
    switch (msg->what()) {
        case kWhatMonitorQueue:
        {
            int32_t generation;
            CHECK(msg->findInt32("generation", &generation));


            if (generation != mMonitorQueueGeneration) {
                // Stale event
                break;
            }


            onMonitorQueue();
            break;
        }
    }
}

void LiveSession::onMonitorQueue() {
    if (mSeekTimeUs >= 0
            || mDataSource->countQueuedBuffers() < kMaxNumQueuedFragments) {
        onDownloadNext();
    } else {
        postMonitorQueue(1000000ll);
    }
}

mDataSource at first with no buffer from internet ,so here triggering onDownloadNext() to download

In Media Framework  :android_src\framework\base\media\libstagefright\include\LiveSession.h


    enum {
        kMaxNumQueuedFragments = 3,
        kMaxNumRetries         = 5,
    };

In Media Framework  :android_src\framework\base\media\libstagefright\httplive\LiveSession.cpp

void LiveSession::onDownloadNext() {
    size_t bandwidthIndex = getBandwidthIndex();


    if (mLastPlaylistFetchTimeUs < 0
            || (ssize_t)bandwidthIndex != mPrevBandwidthIndex
            || (!mPlaylist->isComplete() && timeToRefreshPlaylist(nowUs))) {
        AString url;
        if (mBandwidthItems.size() > 0) {
            url = mBandwidthItems.editItemAt(bandwidthIndex).mURI;
        } else {
            url = mMasterURL;
        }


        bool firstTime = (mPlaylist == NULL);


bool unchanged;
        sp playlist = fetchPlaylist(url.c_str(), &unchanged);
        if (playlist == NULL) {
            if (unchanged) {
                // We succeeded in fetching the playlist, but it was
                // unchanged from the last time we tried.
            } else {
                LOGE("failed to load playlist at url '%s'", url.c_str());
                mDataSource->queueEOS(ERROR_IO);
                return;
            }
        } else {
            mPlaylist = playlist;
        }


        if (firstTime) {
            Mutex::Autolock autoLock(mLock);


            if (!mPlaylist->isComplete()) {
                mDurationUs = -1;
            } else {
                mDurationUs = 0;
                for (size_t i = 0; i < mPlaylist->size(); ++i) {
                    sp itemMeta;
                    CHECK(mPlaylist->itemAt(
                                i, NULL /* uri */, &itemMeta));


                    int64_t itemDurationUs;
                    CHECK(itemMeta->findInt64("durationUs", &itemDurationUs));


                    mDurationUs += itemDurationUs;
                }
            }
        }
    sp buffer;
    status_t err = fetchFile(uri.c_str(), &buffer);


    mDataSource->queueBuffer(buffer);


    mPrevBandwidthIndex = bandwidthIndex;
    ++mSeqNumber;


    postMonitorQueue();
}

onDownloadNext() downloads a media segment list in playlist and queue the downloaded buffer to  mDataSource, then continuously postMonitorQueue() to download next