API questions...

Discussion in 'Public Game Developers Forum' started by cubytes, May 15, 2010.

  1. cubytes

    cubytes Well-Known Member

    Aug 25, 2009
    501
    0
    0
    Hey Devs,

    im curious...

    about the iPod Library and/or playback API:

    -does it give you access to a currently playing track's title, artist, album, genre, album art ect? in other words can an app know what song is playing? by whom? what genre? ect? even if it was playing before the app was launched?

    -is it possible for the an app to recognize the tempo of the track playing? BPM? equalizer analytics? ect? of a track playing in real time? to say be able to synchronize a zooming fractal or a trippy visualizer with a track playing from ipod? or be able to synchronize multiple layers of a visualizers effects with a tracks beat, keys, bass, ect?

    wondering about QuickTime streams/web podcasts (not thru itunes) like sub.fm/iphone (for example)...

    -could an app recognize that there is an active media stream?
    -be able to read/access any tags/meta data that the distributor adds to it like track listing or other info in real time?
    -and be able to recognize the streams tempo? BPM? equalizer analytics? ect?

    if not oh well, but if so i have an idea :)

    which i kind of alluded to in my questions, but it goes much deeper then just synchronous visualizers i assure you....
     
  2. cubytes

    cubytes Well-Known Member

    Aug 25, 2009
    501
    0
    0
    Hey Devs,

    I have decided to go ahead and explain this whole concept out in detail because....

    A. I'm extremely broke and unfortunately I'm also stuck in a huge black-hole of debt (student loans) so I can't really afford to fund/develop this at all (Que the violins please...)

    B. I have no clue how to write code, no idea where to start, and I'm not a good self learner (To be honest). Don't get me wrong I would LOVE to learn how to write code but I would definitely need to take classes (which I can't afford because of A) and be honest wouldn't this whole idea be a prime candidate for a CS class project anyways?

    C. All of the above!

    D. Because I feel like it :)

    Still here? Awesome! Lets begin.....

    Project Name:

    Jam!

    the domain name jam.com looks parked too depending on how cooperative the guy who owns it and the name this could work well with marketing (logo/promotion) and be really easy to remember. if not oh well a name is just a name...

    Synopsis:

    develop a simple mobile application (for the iDevices, Android, and WebOS ect) and a webapp (.com) to do the following:

    -fetch contextual data like relevant blogs/news, album art, photos, random video clips, music-video clips, lyrics, an other contextual data to music/tracks playing in the background

    -use trippy visualizers, slideshow animations, and visual effects (that at least sync with the music playing on some level)

    -sprinkle on some social functionality like comments (via anonymous/guest post, disquss, open id ect) and of course leverage third party social networks a la the standard facebook and twitter integration

    Thoughts:

    I know what you're thinking, "there's already applications that have this functionality, why bother?"

    and yes you're right... kind of.

    The 'iTunes LP' does offers extra content (albeit static) with your music, but you have to pay extra $$$ for it and you *must* use iTunes.

    The XBOX360 'Last.FM app' also provides a rich slideshow presentation experience (thats kind of inline with what i want to do) with high quality photo slideshows that play while you listen to your music, but you have no control over what tracks play all you can do is create a station by artist or song and skip, plus you have to endure random adverts.

    and last but certaintly not least songbird a cross platform open source media player that comes with the mashTape add-on installed by default and yes it does fetch contextual data but yet again you *have* to use songbird for it to work, and keep in mind there may also be more plugins for songbird that do kinda what I'm proposing too...

    as far as mobile goes im not aware of any application that does what i am proposing here at the moment.

    Now here's the pitch....

    The secret sauce to this idea (if you will) is that I want it to be media player independent and not intended to be *the* media player at all and it's not going to be just another plug-in either. In fact the idea is that the user doesn't have to switch media players or re-configure their library or none of that. All they have to do is play the song/track/station they want to hear on their preferred media player (or open an audio stream via the web) then launch/open the 'Jam application' if they want too, which will then do what I proposed above. it's as simple as that. no extra cost to the user, its not a plugin for a media player that complicates things its just very simple and easy to do.

    For the mobile smart-phone and smart-tablet(?) platforms especially the iDevices the Jam app would depend heavily on the APIs available from the media player of the device, and some really smart sync options for the users to cache the contextual data locally so they can still enjoy the experience even when they have no signal/connection. The APIs needed will be:

    -access to the track-listing info, ID3tags and other metadata attached to the audio file

    -access to genius results and playlist data

    -be able to run analytics of audio playback. so you can synchronize visualizers, animations and visual effects (of a slideshow) with the audio playback. I also want to be able to sync multiple layers with multiple elements of the track playing. such as; an effect that syncs with the beat/drum-patterns/tempo, while another effect/animation simultaneously syncs with the vocals of a song or an epic rift perhaps, with yet another effect/animation syncing with the bass (all at the same time)


    For the PC platform (desktop, laptop ect) the question is: can a web-application (for starters) access the data needed to pull this off across all the popular OSes? ideally (or down the road) it would be wise to also provide an application that can be locally installed on the machine but i want to try and pull it off completely with just a webapp alone first and foremost where no installation is needed at all (besides maybe a browser plug-in for application data? or something)

    For the TV/stereo systems I would definitely be EXTREMLY interested on how to get this on the Google TV and the game consoles (XB360 PS3 Wii OnLive) so that it can provide the same experience weather you're:

    -watching a music channel on dish network (or other network providers)

    -playing/streaming audio on any of the game consoles

    -listening to music on your stereo system weather it be a CD player, am/fm-radio/hd-radio/sat-radio/internet radio, or a media device (assuming the user's stereo system has a hard-drive too)

    Feature Set

    PRIORITY............

    front facing:

    -a full screen *rich* multimedia slideshow with effects and animations that sync with the music playing in the background using contextual data

    -types of photos I want; album artwork, photos of the band, themed images that are relevant/emphasize/go well with the genre of the music playing

    -fetch album art from ID3 tags & fetch photos from online sources (flikr, facebook, picasa ect)

    -videos: official music video (if applicable), random videos from live performances of artists of the track playing in the background, or other video types that go well with the genre of the track playing in the background (basically epic youtube video mash-ups)

    -mix in and sync these video clips (no audio) with the slideshow seamlessly

    -start of with a standard set of trippy visualizers, and effects that seamlessly mix and go along with the slideshow of photos and video clips and expand them periodically via updates (backend & frontend)

    -toggle lyrics on/off button (they should be presented like a rich karaoke machine) a la lush font, and text animations that sync with vocals

    on the back:

    -there will be flip/info button located somewhere (maybe top right). it will essentially flip the window around (when clicked or tapped) thus reveling the other contextual data and the social features i mentioned above on the back. just click/tap it again to flip the window back around thus resuming the slideshow

    -on the back there will be; a top section with album artwork and proper track-listing info (track tittle, artist, album, genre)

    -under this will be 5 tabs; 1) biography, 2) news, 3) touring info, 4) comments, 5) similar

    -bio: generate a rich bio via wiki (duh) and social network profile info (if possible)

    -news: fetch activity from social networks, and popular blog listings via digg or other crowd sourced news aggregates

    -tour: just list out concert dates and ways to get tickets (for starters)

    -comments (both local comments attributed directly within the app, along side comments fetched from 3rd party networks (twitter, fb, ect) relevant to the track song thats playing

    -similar artists/tracks: list of similar artists/tracks by genre, or better yet via music genome project (down the road this tab can also include crowdsourced data too)

    sub features:
    -rating system (thumbs up and down)
    -can accumulate this crowd sourced data and tie it back into the similar artist/track tab to improve music discovery
    -might need to come up with a solution to make it easier for users to sample the music from the similar artist tab. i like the last FM ratings list that show all the tracks filtered by the most popular (would be nice if they had an open API for that) from each artist but i would like to be able to sample more of the tracks, even if it is just a 30clip and of course fetch relevant youtube videos too
    -i would like to maybe expand the tour tab so that each concert/venue after the fact will also be an aggregate or a profile of the concert/venue where users can upload/attribute content like videos from their cameras, photos that they took, and lots of comments (integrated of course with 3rd party networks)

    and thats its :)

    if you have any questions, feedback or anything don't hesitate to let me know. feel free to reply in this thread or if you want to contact me privately hit me with a pm and i will share my contact info with you.
     
  3. cubytes

    cubytes Well-Known Member

    Aug 25, 2009
    501
    0
    0
    on a personal note..... que the violins ;)

    I feel strongly that this idea has the potential to be hit (i cant guarantee it though), as in hit critical mass 500,000+ downloads easily (give or take a few hundred) and possibly offer up a great exit strategy down the road. Sure there are some *what if's* that i can't account for at the moment since I don't know much about the APIs in question (hence the reason I started this thread in the first place) or weather or not Apple will simply reject it (hopefully not) but I assure you this idea definitely has merit....and im sure a few of you feel that way too :)

    Im also not going to lie... Im pretty much screwed financially for the better part of the rest of my life. My family has always been on the brink of poverty (i still love them a LOT though ;) and I was desperate situation so I took loans to get into a poor mans ITT tech school (keiser college) and ended up having to drop the course because I couldn't afford the tuition near the end (I was sooo close too). Now I'm still broke, my family is still broke (and yes i still love them a LOT) and voila now I have this huge black-hole of debt hanging over my shoulders with no opportunity in sight to dig myself out, fast enough to be in a viable position to do something with my life and be a part of something that I can be proud of.

    On the flip side though I've always aspired to be an entrepreneur. I'm passionate, I have vision/ambition, Im a fast learner, and my imagination knows no bounds. I know many of you developers believe ideas to be worthless (I understand where you're coming from of course) but to be honest ideas is about all I can afford to do at this point in my life so to me my ideas are my only hope to make something spectacular happen :)

    sorry for the personal stuff... but i felt i had to elaborate a bit as to my reasons/motivations for posting ideas on various forums (this one included).... it's for the sake of hope i post my ideas ;) although i never really get my hopes up... ironic isn't it?
     
  4. GregH

    GregH Well-Known Member

    Sep 12, 2009
    479
    0
    0
    Wow, this is ironic given your posts in the various piracy threads.
     
  5. cubytes

    cubytes Well-Known Member

    Aug 25, 2009
    501
    0
    0
    and your point is? ;)
     
  6. bmn02102

    bmn02102 Well-Known Member

    May 24, 2010
    144
    0
    0
    I believe there's no way for any application to access the raw sound output/data, because the decoding is handled by the hardware.

    Unless you really want to manually transcode every song to WAV before playing it (which would take a lot of time and storage space, and kill your battery pretty fast), you're going to have to rely on either line-in or mic inputs (mic should be fine since you only need mono), but either way you're probably going to require extra hardware.
     
  7. cubytes

    cubytes Well-Known Member

    Aug 25, 2009
    501
    0
    0
    thx bmn02102 thats good info... much appreciated :)

    hmm so it sounds like syncing visualizers, slideshow animations, effects, and transitions in real time is out of the question, bummer that was a key feature too...

    any ideas for a work around?

    as in scanning the file to extrapolate patterns that can be used for the sync? or caching the patterns in the back-end on the sever side and then matching up the time/scrub of a track playing in real time?
     
  8. DropDKeith

    DropDKeith Well-Known Member

    Unfortunately, Apple does not give you any way to access the song data at all. You cannot access the files or the bytes. You can only query for metadata such as the artist/song info and album art.
     
  9. bmn02102

    bmn02102 Well-Known Member

    May 24, 2010
    144
    0
    0
    Nah, the problem runs even deeper than that. Aside from the sandboxing issue which is actually the least of your problems, you'd still be stuck having to implement your own MP3 decoder in software no matter how you went about things, as well as needing the space to store the resulting WAV (which is typically about 10 times larger than the original MP3).

    To put things in perspective: importing an MP3 into a waveform editor on my PC (3.0Ghz core duo & 4GB RAM) takes about 20 seconds. Now I don't know about you, but if I had to wait even that long for my media player to start playing a song, I wouldn't use it; I'd switch to a different one. By comparison, the iphone is about an order of magnitude less powerful than my PC.
     
  10. cubytes

    cubytes Well-Known Member

    Aug 25, 2009
    501
    0
    0
    thx guys :) i figured as much...i mean it could still fetch contextual data with the track-listing alone, although i would prefer to go at it from multiple angles but it could still work (at least).

    As far as synchronous slideshow/visualizer animations, effects though, especially the multi-layer sync idea, that seems pretty much out of the question on the iphone, if I cant even access or tell how long the song has been playing in real time.

    i guess the only other option would be to build out a database-farm and cache patterns that will be used for the sync of the slideshow animations, visualizer effects ect to the music for each and every song (known to man) on the back-end and just send that to the device when the user keys up a song. If they start the song before launching the app just have a brief reminder to restart the song/track to have the best experience or something like that....or let them interact with the slideshow as in be able to fastforward or rewind it with a timestamp that will make it easy for the user to manually sync it if they want too


    but i think it should go without saying that such a database would take a while to build out and cost a LOT of $$$$
     
  11. DropDKeith

    DropDKeith Well-Known Member

    The current play time is one thing that you CAN get. :)

    Though different rips of the same song can have different lengths (different intro and outro timing) just to keep things fun.

    Not to mention that all the song meta-data varies wildly in quality and accuracy...

    Good luck. :)
     
  12. cubytes

    cubytes Well-Known Member

    Aug 25, 2009
    501
    0
    0
    Oh niiice thats good news!! thx for the info DropDKeith :)

    The sync is a priority feature and i wouldn't be above building out databases of patterns then hashing them and pushing that to the devices using the current play time as a guide of sorts.

    I know it wont be perfect, and come with high overhead of cost and the cost of acquiring access/right to scan the tracks for patterns in the first place but it's better then just ignoring the feature all together. As the feature itself plays a crucial role in the overall experience for the user and it is *the* feature that will set Jam (or whatever name that ends up being used) apart from anything else available at the moment.

    its that important :)

    without the sync its just a slideshow with the sync its whole new kind of experience. plus its challenging to pull off (as you and other devs alluded to in this thread) so its the only thing that will set Jam apart from the flood of competition... i like to think ahead ;)

    also with being limited to just the tracklisting, theres also the challenge of how to improve the relevancy of the contextual content. i think the key for relevancy is a stacked approach; utilizing multiple tools to verify, and of course establishing partnerships with reliable & cooperative sources of content rather then relying on whats the easiest, quickest and simplest way to get the content...
     

Share This Page