Showing posts with label graphics. Show all posts
Showing posts with label graphics. Show all posts

Friday, June 27, 2014

Google I/O 2014: Rehash

All of the videos have been posted from the various sessions I was in this year. Here they are, along with links to the slides.

What's new in Android

A presentation with +Dan Sandler that provides a quick overview of some of the larger features and new APIs in the L Developer Preview release and other new bits in the recent Androidosphere. There's also a really good deep-dive into Notifications, since Dan's the non-local expert on the subject.



Slides (PDF)


Material science

This session, presented with +Adam Powell, provides an overview of the engineering side of the Material design system. Many of the other sessions at Google I/O this year discussed the design side; this presentation covers the technical details of the APIs and the capabilities exposed by the framework for Android developers.



Slides (PDF)


Material witness

I was happy to be joined once again by former Android developer and UI Toolkit team lead +Romain Guy for this talk on some of the new Material design elements in the L Developer Preview release. The idea behind this talk was to go a bit deeper into using and explaining the new APIs, as well as explaining how some of the features, like realtime soft shadows, work.



For slides as well as demo code, check out Romain's blog at curious-creature.org.


Android fireside chat

This session, organized and moderated by +Reto Meier, I found to be more interesting than other such panels I've seen. Often, these things tend to have a lot of awkward silences as the panelists try to figure out the most interesting way of saying "No comment" since there's a general policy on Android of not talking about future development plans. This time, there was a lot of discussion around how and why some parts of the system work, which I enjoyed as an audience member that just happened to be sitting somewhat closer to the panel.

Thursday, July 25, 2013

New in Android 4.3: ViewOverlay

Since we just released Android 4.3 yesterday, I thought I'd wax poetic about one of the features I worked on in the release: Overlays.

Android devs
Come out to play:
Use the new
ViewOverlay.

There are many ways to get custom graphics (drawables) to appear in a view. You can set a background drawable if that's what you want, or you can use an ImageView, or you can create a custom View subclass and override onDraw(). Or if you want to draw them over the children in a layout, you can override the layout and override dispatchDraw() to draw them after all of the children (after a call to super.dispatchDraw()). But sometimes you just want something simpler: ViewOverlay.

Here's how it works:

You call View.getOverlay(), which returns a ViewOverlay object. Then you call add(Drawable) on that overlay object... and that's it. You can also call remove(Drawable) to get rid of the drawable or clear() to get rid of everything in the overlay, but that's the entire API. The only other detail is that you need to set the bounds on your drawables (as you would if you were drawing them manually in onDraw()); the overlay doesn't do any positioning or sizing, it simply draws everything you tell it to, in the order in which the objects are added.

Oh, and one other thing: you can also add View objects to an overlay if you get that overlay from a ViewGroup. That is, ViewGroup overrides getOverlay() to return a ViewGroupOverlay, which is a subclass of ViewOverlay that also deals with views in addition to drawables. Once again, the API is simple: there's an add(View) method and a remove(View) method. And once again, you are responsible for positioning/sizing the views where you want them in the overlay; the overlay does not do any layout.

Essentially, the overlay is a container for drawables (and sometimes views) that is the exact size of its host view. It doesn't do layout; it just draws the stuff its told to, after it's drawn everything else in its host view (including any children of that host view, if the view is a ViewGroup).

Important caveat, especially with respect to ViewGroupOverlay: The overlay is intended for temporary display of views, not for any long-term or functional use for views. In particular, the views there are not laid out by the container (as noted above) and do not participate in focus or input events. Overlays were created to be a visual-only mechanism, for things like transient animations. Think about things like fading out a view. This can be tricky in some situations where removing it immediately removes it from the view hierarchy and makes it tough to have it drawn while it's fading out. Now you can pop it into the overlay of its parent while it's going away.

Here's another interesting thing to ponder: you can add drawables/graphics anywhere in the view hierarchy that makes sense for your situation. So, for example, if you wanted to fade out a view or some drawable while sliding it off the screen, or up-scaling it, outside of its parent container, this might normally be a problem because parent containers like to clip their children to their bounds by default. Now you can add the object into an overlay anywhere up the tree, to give you the flexibility to move/resize it as you see fit without worrying about the clipping constraints of that original parent.

Anyway, I'm sure you can find interesting things to do with overlays. So go ahead!

Friday, May 17, 2013

Google I/O Presentations Now Live

Romain Guy and I gave two talks at Google I/O yesterday which are now live on YouTube:

Android Graphics Performance: http://www.youtube.com/watch?v=vQZFaec9NpA



Slides for Android Graphics Performance are posted on Romain's blog.


A Moving Experience: http://www.youtube.com/watch?v=ihzZrS69i_s



A Moving Experience Slides:

The code from the animation demos will be posted soon, probably in conjunction with future DevBytes episodes.

Friday, February 8, 2013

Thursday, January 31, 2013

DevBytes: BitmapAllocation

This example shows how to speed up bitmap loading and reduce garbage collection by reusing existing bitmaps.

Note that the reused bitmap must be mutable and of the same size and configuration as the bitmap you load into it. Not sure if this is clear from the video, although it is documented in the API.

Code: http://developer.android.com/shareables/devbytes/BitmapAllocation.zip

Video: http://www.youtube.com/watch?v=rsQet4nBVi8

DevBytes: BitmapScaling

This example shows how the use of BitmapOptions affects the resulting size of a loaded bitmap. Sub-sampling can speed up load times and reduce the need for large bitmaps in memory if your target bitmap size is much smaller, although it's good to understand that you can't get specific Bitmap sizes, but rather power-of-two reductions in sizes.

Code: http://developer.android.com/shareables/devbytes/BitmapScaling.zip

Video: http://www.youtube.com/watch?v=12cB7gnL6po

Wednesday, January 16, 2013

DevBytes: Short, Focused Android Tutorials

The Android team has just rolled out the first four shows in a new series called DevBytes, which is intended to be very short, focused tutorials on specific developer topics. A quick explanation, maybe a demo, and a few lines of code to walk through and it's done. Taking a break from work? Want to learn something new? Or just wonder how a specific piece of technology works on the platform? Tune into one of the DevBytes episodes to get a quick deep dive into that exact topic (well, assuming we've done a show on that topic. Maybe we could interest you in one of the shows that we've actually filmed yet instead?)

The first few of these feature ... me! (Surprise, surprise!) I'll be doing many of these tutorials over time, especially in the areas of animation, graphics, and UI development, starting with this first set:


ViewAnimations:
This show covers the basics of creating and running pre-3.0 Animations on View objects.

PropertyAnimations:
This show parallels the ViewAnimations demo, but uses the new Animator classes introduced in Android 3.0.

WindowAnimations:
Ever want to use a custom window animation to transition between sub activities? Watch this show.

CrossFading:
TransitionDrawable can be used for very simple cross-fade effects between two different Drawable objects.

There will be other shows coming online in the future as well; we're just starting with some of the ones that I did. I wouldn't mind if everything were about animation, but that's just me...

There are code links in the video descriptions if you want to see the code I cover in the videos. I'll also be posting individual blogs with the video and code links to make this easier, I just wanted to get this introduction/overview out there to get it rolling.

These are just a start: we have more videos on the way in the coming weeks. So tune in and geek out.

Thursday, August 23, 2012

Thanks for the Memories

(This tip and anecdote is specifically about Android, but the same technique applies to every platform I've ever worked on, which at this point is quite a few. In fact, it's a technique that Romain and I have stressed in our book and in most graphics/GUI talks that we've given over the years, such as this one from Devoxx in 2008) .
I was debugging an application recently (names withheld to protect the completely and utterly guilty) and discovered that the source of a serious performance bottleneck was simply the size of the bitmaps involved.

The application's job is to display lots of pictures, so using bitmaps is a given. And the size of the bitmaps being loaded and displayed is signficant, so there were going to be issues around memory and performance anyway. But it was the way in which the application was treating the source and destination sizes that was at the root of the problem.

In particular, the application was loading each image into a bitmap of size X x Y. Meanwhile, they wanted to draw that bitmap into a destination rectangle half that size, .5X x .5Y. This is easy to do; you just call the appropriate Canvas.drawBitmap() method with the relevant source/destination rectangles, or specify a scale on the Canvas object, and we'll take care of the details.

Simple.

But.

Wrong.

That is: Scaling on the fly is easy, and it works. But asking us to do your work for you on every frame in which you draw that bitmap might cost you significantly in performance and memory when there's a very easy way for you to do this once and simplify all future operations with that bitmap.

Here's the right thing to do: pre-scale the bitmap to exactly the size you need. Then when you need to copy it into the destination, you call Canvas.drawBitmap(left, top, Paint) (the version that doesn't take a dest rectangle) and then all that Android needs to do is copy the bitmap. Much simpler. And what's more: it requires potentially far less memory than downscaling to a smaller size.

This is probably (hopefully? please?) obvious when you're running in a software-rendering situation (e.g., all releases prior to 3.0, or any app targeting pre-4.0 releases and not specifying hardwareAccelerated="true" in the manifest); having the framework scale the image every time it's drawn means going through a much slower path than a simple 1-to-1 copy entails.

But what about on GPUs, with our wonderful new hardware-accelerated world of Android apps as of version 3.0+? Aren't GPUs supposed to be faster at stuff like this? What are we paying them for, anyway?

Yes.

But.

Here's the problem: the actual scaling operation is quite cheap on a GPU, even negligible. But that's not all that you should be concerned about as a mobile developer. Mobile developers should always worry about memory. You should profile your application. You should think about memory consumption at night when you can't sleep. You should bring it up on first dates*, and fester on it while on vacation.

If you're displaying several images per frame, you want to be very aware of how much memory those bitmaps are soaking up. This is true for the bitmaps in CPU memory, but also true for bitmaps that we upload to textures. Just because it's cheap for a GPU to scale a large texture into into a small space on the GPU doesn't mean it's fast to upload it into texture memory, or cheap for the GPU to have that large image sitting around in memory. Memory is a constrained resource and should be treated as such. I'm sure your date will tell you as much (possibly as they leave the date in search of more interesting prospects).

If you're going to scale from a 2k x 2k image into a 32x32 icon, wouldn't it make more sense to pre-scale it once, chuck the original one, and thenceforth deal with only the smaller version instead?

Of course, if you really need full-size images, then go ahead and do what you need to do. And if you're animating an image's size (such as zooming in on it), then pre-scaling to each intermediate size probably doesn't make much sense. But if you know that you're going to be using a smaller version for a while, then you should probably pre-scale to that size rather than drag around the memory and performance baggage associated with the original version, no matter what the hardware acceleration situation on the target device is like.

* The advantage of discussing memory consumption on first dates is not only that it will help you keep it in mind at all times, but also that this will inevitably lead to more first dates on which you can continue discussing it. Or it will at least result in less second dates.

Tuesday, July 10, 2012

Google I|O 2012 Presentations

The videos and slides are now posted for the presentations that I did with Romain Guy at Google I/O a couple of weeks ago.

What's New in Android: A developer's overview of some of the new features available in Android 4.1 Jelly Bean, including a deep dive into the enhanced Notifications capabilities (this part by Daniel Sandler, one of the engineers responsible).

For Butter or Worse: A very geeky look into how hardware accelerated rendering works on Android, what we did to make things smoother and faster in the latest release, and what Android developers can do to make their applications faster and smoother as well.

These videos were posted within a day of the talks, but the slides took a bit longer (possibly explained by the rush up to IO followed by everyone collapsing in fatuge when it was over).

There are tons of other great talks from the conference that are posted as well, both in the Google Developers channel on YouTube and (with slides) on the IO site itself.

Enjoy...

Wednesday, January 4, 2012

Curved Motion in Android

The animation support added in Android 3.0 and enhanced since then is useful, allowing a flexible system of property animation to animate literally anything you want. Like a store going out of business: if it ain't nailed down, it's up for grabs.

But that doesn't mean we're done yet; there are many things that we'd like to do to keep improving the animation system and making Android animations easier to write and more powerful. One of those things is curved motion.

Right now, if you animate something between a start and end point, it will move in a straight line between those values. For example, a translation animation, where the object moves from one (x, y) location to another, will move in a straight line beween those points. You can (and should) use non-linear timing for these animations, and the animation framework uses ease-in/ease-out timing by default. But the actual spacial movement is linear; there is no way for you to specify a curved path for an object to take between the end values of an animation.

Even if you use a multi-point keyframe animation, where you specify several intermediate points for the animation to pass through along the way, you are still going to move linearly between each of those points.

One goal with animations is to make them feel as natural as possible. And just as motion in the real world (you know, the one we have to use as we move between the machines of our lives) is not linear, animations in our GUIs should not be limited to linear. For example, f a view moves from corner of the screen to the opposite corner, wouldn't it be nice to have the option to give it a little curve in and out at both ends, instead of being locked into a straight-line movement?

We'd like to add this kind of capability to the animation system in a future release; it needs to be easy to create such animations, we just have to provide the APIs and functionality to let you do it.

As I was looking into the problem, I created some prototype code using the existing animation APIs and realized that there's nothing stopping you from having this capability already. The flexibility of the Animator APIs allow you to do exactly the kinds of operations you need to get curved motion. You just need a little more code to do it.

I thought it would help to post some of my prototype code to show you how. In particular, I thought this sample was interesting because it shows:
  • How to move things along curves and complex paths
  • How to use the Animator APIs to do more than what the base library supports. In particular, how to use TypeEvaluator to animate custom types.

Some notes and caveats before we begin:
  • This is a prototype only, and does not necessarily represent the way it would appear in any final API. It's just a sample program, and a pretty simple one at that.
  • Simply computing the location based on the fraction time elapsed in a curve interval is probably not the motion you want. It will give the mathematically correct motion along that path, but the time spent traveling along any particular length of that curve is dependent on the structure of the curve. Basically, you'll end up with slower and faster portions of the curve. This problem is admirably described on this blog. A more complete solution flattens the curve and ensures uniform speed. But again, it's just a simple demo, so I'll leave correct path-time-distance navigation as an exercise for the reader (and for the writer, since this would be a part of any future implementation in the SDK).
  • The timing of the animation along a multiple-point path such as the one in the demo app is not as flexible as I'd like it to be. Basically, you end up with something that gives equal time in the overall animation to each individual interval. In addition, any "move" operations in the middle of the path cause the animation to wait at that location for that interval's equal slice of the duration. It should be possible, in a more complete implementation, to define the time associated with any particular interval.
  • This description assumes a foreknowledge of Bézier curves; if you have no idea what I'm talking about, you might want to go look them up (such as on the blog linked above or various other places on the web, such as Wikipedia). Or you can just read along, refer to the mathematically imprecise sketch to the right, and hopefully not get too lost.
  • The code as written requires Android 4.0 to build. Actually, it's mostly compatible back to 3.0, but the PathEvaluator class uses a generic specification for TypeEvaluator that was introduced in 4.0 (not necessary, just convenient when I wrote the code).

On with the code.

The activity code is in PathAnimationActivity.onCreate(). First, we set up the path itself:
AnimatorPath path = new AnimatorPath();
    path.moveTo(0, 0);
    path.lineTo(0, 300);
    path.curveTo(100, 0, 300, 900, 400, 500);
Here, we are constructing an AnimatorPath (which is part of the demo project that we'll see below) and supplying it with operations that will become points in the path, along with the operations to navigate the intervals up to those points. The first operation defines where the path starts, (0, 0). Then we move in a straight line to (0, 300). Finally, we move along a curve (a cubic Bézier curve, to be precise) to the point (400, 500), using control points (100, 0) and (300, 900) along the way.

Next, we set up an ObjectAnimator to animate this path:
    final ObjectAnimator anim = ObjectAnimator.ofObject(this, "buttonLoc", 
            new PathEvaluator(), path.getPoints().toArray());
This animator uses a new PathEvaluator object (introduced below). It also queries the AnimatorPath object to get an array of PathPoint (covered below) objects; these will become the points in the animation that define the intervals that we animate between. The animator will send the animated values to the this object, which is the activity instance itself. We implement the setter below to receive those values and pass them along to the actual Button object that we want to move on the screen:
    public void setButtonLoc(PathPoint newLoc) {
        mButton.setTranslationX(newLoc.mX);
        mButton.setTranslationY(newLoc.mY);
    }

The AnimatorPath class referred to above stores information about the overall path. Its API consists of everything seen above:
public void moveTo(float x, float y);
    public void lineTo(float x, float y);
    public void curveTo(float c0X, float c0Y, float c1X, float c1Y, float x, float y);
    public Collection getPoints();

Internally, AnimatorPath uses PathPoint to store the information at each point along the path. The PathPoint class is a simple data structure that just holds an x/y location, optional control point information (for curves), and the operation that tells the path containing that path point how to nagivate the interval leading up to that point. There are three factory methods that AnimatorPath uses to construct PathPoints as its API is called:
    public static PathPoint moveTo(float x, float y);
    public static PathPoint lineTo(float x, float y);
    public static PathPoint curveTo(float c0X, float c0Y, float c1X, float c1Y, float x, float y);

All of the logic of actually animating between points along the path (besides that in the Android Animator engine itself) is in the class PathEvaluator. This class implements the single method in the TypeEvaluator interface, evaluate():
    public PathPoint evaluate(float t, PathPoint startValue, PathPoint endValue) {...}
The return value of evaluator() depends on the operation described by the endValue PathPoint object.

For curves, we calculate the x/y values given the anchor points (the location at startValue and endValue) and control points (both control points are stored in the endValue structure).
    if (endValue.mOperation == PathPoint.CURVE) {
        float oneMinusT = 1 - t;
        x = oneMinusT * oneMinusT * oneMinusT * startValue.mX +
                3 * oneMinusT * oneMinusT * t * endValue.mControl0X +
                3 * oneMinusT * t * t * endValue.mControl1X +
                t * t * t * endValue.mX;
        y = oneMinusT * oneMinusT * oneMinusT * startValue.mY +
                3 * oneMinusT * oneMinusT * t * endValue.mControl0Y +
                3 * oneMinusT * t * t * endValue.mControl1Y +
                t * t * t * endValue.mY;
    }
For lines, we perform a simple linear interpolation between the start and end points:
    else if (endValue.mOperation == PathPoint.LINE) {
        x = startValue.mX + t * (endValue.mX - startValue.mX);
        y = startValue.mY + t * (endValue.mY - startValue.mY);
    }
For moves, we simply jump to the location defined by the end value:
    else {
        x = endValue.mX;
        y = endValue.mY;
    }
Finally, we create a new PathPoint with this calculated xy information, which will be passed to the setButtonLoc() method seen above:
    return PathPoint.moveTo(x, y);

... and that's it. There's really not much to it, which was the whole point of posting this code. If you want to add curved motion to your animations, you can wait for the framework to do it for you... or you can take what I've done here and modify it to suit your needs. Yay, TypeEvaluator!

I've posted the code on code.google.com; check it out for yourself. Or you can download the Eclipse project (including the pre-built apk) here.

Saturday, November 5, 2011

Devoxx: Then and Now

I'm starting to prepare presentations for the ~8 hours of talks that Romain and I are giving at Devoxx in a couple of weeks. "Preparing" generally entails mostly worrying, followed by a mad rush of writing slides and code at night, on the long flight, after beers, between talks, and sometimes during the talks. It's a busy time of year.

I realized that the organizers of Devoxx had recently released all of the recorded talks from last year into the wild (read: they're free on parleys.com), so it seemed worth linking to them in case anyone wanted to see what we had to say last time around.

I'll give a plug for the conference and the Parleys site here. If there were an ad banner, it'd go here. Not because I'm paid (I would make a poor ad salesman, apparently), but because I think that both the conference and the parleys site rock. The organizers do a great job of putting it all together, and the recordings and presentation of the talks on parleys.com is the best I've seen by far of any conferences I've spoken at.

Yes, that young man on the pedestal is
throwing a hand. It's an Antwerp thing.
Parleys is a subscription site; after the annual conference, you can join for the year (for a fee of 79 Euros) to watch all talks given at the conference. During that year, the talks are made freely available, one by one, then at the end they are all free. That year has passed, so the 2010 talks are all available now for free. If you enjoy them, consider joining parleys next time around to see them earlier. Or better yet, join us in balmy Antwerp, where the beer is good and so are the fries.

Dive into Android, Part 1
Romain talks about layout. This talk includes live-coding a custom layout, which is a good lesson in how to do it yourself (although you probably don't need to do it on stage in front of the cameras to make it work the way you want).

Dive into Android, Part 2
I talk about custom graphics. This is kind of a Filthy Rich Client talk, but more focused on the core principles and approaches of doing custom graphics in Android applications. nothing over-the-top filthy, just good stuff to know about Android GUI development.

Android Graphics and Animations
We cover lots of architectural details about the Android platform, including the view hierarchy, classes used in custom graphics, and pre-3.0 animations (remember: this talk was given before Android 3.0 (Honeycomb) was released).

Android UI Development: Tips, Tricks, & Techniques
This talk consists of a smattering of tips that will help you write efficient Android applications. We discuss the various tools that you should use, techniques for avoiding unnecessary garbage creation, and various performance tips.

Flex 4 FunThis is my swan song for the platform that I worked on prior to Android, and for my Flex 4 Fun book that was published last year. I cover various things from the book, including graphics objects, filters, states and transitions, component skinning, and (of course) animation effects (the area I worked on for Flex 4). Lots of demos and code.

Thursday, August 4, 2011

DroidFlakes: An Android Animation Demo

Here's a simple app that I wrote recently to see how background processing affects frame rate. There's nothing too complicated going on in the app, but it seemed worth publishing to show how to do various things in Android, such as:

  • Using a single ValueAnimator (3.0+ API) to animate several objects independently
  • Manipulating the transform of Canvas to move/rotate objects
  • Using the GPU to provide faster rendering (3.0+ capability)
  • Tracking the current frames per second performance
Here's a video of the running application, captured using HDMI on a 3.1 tablet. You can see how the More, Less, and GPU controls affect what's going on in the app. Note that it starts out with no GPU acceleration and bogs down when we have lots of flakes on the screen. Then see how the frame rate improves once the GPU box is checked. The app is accelerated by default; non-GPU rendering is performed by setting a software layer on the view, which forces rendering to happen in software to a bitmap, which is then copied into the view.

Here's the code (a zipped Eclipse project for the app). You should be able to build/run it for any 3.0+ target device.

Note: this is not the Renderscript app that I showed in a recent user group talk. That other app is based on this one, but uses Renderscript to animate and draw the droidflakes. This version just uses the standard Canvas API to draw bitmaps. I'll post the Renderscript version soon.

Thursday, June 23, 2011

Android Rendering Options

I've given some talks about Android graphics recently, and the same questions keep coming up:

  • What is Renderscript appropriate for?
  • Is it a replacement for OpenGL rendering?
  • How do I access Renderscript from NDK code?
  • When do I choose between the different rendering options?
  • Can I have a free tablet?

It seemed to me that a brief article might help. In particular, I wanted to write something that lays out the various rendering options that Android provides and describes how we envision developers using them (and how we use them ourselves in the applications that we ship) so that you might better understand how and when to use them in your applications. Of course you're not bound to what we think; if you come up with new ways to use what the platform provides, all the better. But we can at least provide a little guidance to help you navigate the myriad of options. Hopefully this article will help answer the questions above. Except for the last question about the free tablet - I can't help you there.

First, I should explain what I mean by rendering, for everyone reading this that's not a graphics geek. "Rendering" is a synonym for drawing. Rendering is also defined as the process of turning an animal carcass into fat and other component pieces... but that's not what I'm talking about here. This article is about is all of the ways, in the Android platform, of drawing things onto the display. For example, you might have a drawing application, or a photo album, or a game, or a reading program, or anything else that needs to draw its content to the screen. If you're just using the SDK components (ImageView, Button, TextView, and so on), then we handle the rendering of these components internally. But if you have custom components that draw their own graphical elements, or if you're writing a fullscreen game that draws everything itself, then you need to know how to draw those items onto the device screen so that the user can see and interact with them.

There are two options for writing Android applications, each with their own ways of rendering graphics. I'll cover each of these in the following sections.

  • SDK
  • NDK

The Android SDK

The SDK is the typical, default approach for most GUI applications. You write your application using the Java programming language and the libraries that the SDK provides. Most of the graphics on the screen are drawn by the framework for the standard View objects such as Button, TextView, ListView, and so on. If you create custom Views or have customized rendering that you want to do for your application, then you need to know about the rendering options available to SDK applications. There are basically three options:

  • The Canvas API
  • Renderscript
  • OpenGL wrappers

The Canvas API

What I mean by "Canvas API" is the standard rendering for a typical SDK application that uses View objects (standard and custom) and performs and rendering by calling the various methods in the Canvas class. The rendering process of the Android view hierarchy consists of calls to each View's onDraw() method. This method takes a single parameter, Canvas, which is the object used by the view to draw its content. For example, a Button might tell its background Drawable to draw itself into the Canvas and then draw its label with a call to Canvas.drawText().

If you have a custom View, in which you implement the onDraw() method and draw your own graphic elements, then you would typically call methods in the Canvas object passed in to the onDraw() method. For example, you might draw lines with Canvas.drawLine() and circles with Canvas.drawCircle().

This approach to rendering is typical for most GUI applications, and is fine for most purposes when you just need standard GUI components and graphics and for which the performance of your application is not an issue. With the GPU acceleration that we added in Android 3.0, the performance for the Canvas API is even better than it used to be. As discussed in the article Android 3.0 Hardware Acceleration, all rendering done by the SDK now happens via OpenGL calls to the GPU. This level of hardware acceleration for standard as well as custom views and graphics can provide excellent performance and allows for parallelism by offloading rendering work from the CPU onto the GPU.

But sometimes you may want to go beyond what the Canvas API provides, either in terms of the graphics functionality you need or the performance that your particular situation requires. For example, the Canvas API does not provide complete 3D rendering capabilities. In these situations, you might want to look beyond the Canvas API to the other options available in the SDK.

Renderscript

The Renderscript API was introduced in Android 3.0. Renderscript was actually in use in an earlier form long before this release; the Live Wallpapers that shipped with earlier releases were written using Renderscript. But Android 3.0 introduced the first public form of the API and the runtime library. Describing what Renderscript is and how to use it is beyond the scope of this article. Instead, I'll just refer you to Jason Sams's introductory articles, Introducing Renderscript and Renderscript Part 2. Very briefly, Renderscript is a language, API, and runtime library that you can use to achieve high performance rendering and computation for Android applications.

An application using Renderscript is an SDK application, using any of the SDK APIs and facilities provided at that level, with addition pieces of the application written using the Renderscript language. These scripts are initialized from Dalvik code. The scripts automatically provide glue code that wraps fields at the Renderscript level, so that SDK code can set values in Renderscript code by calling simple methods. Renderscript has a reference to a drawing surface that it renders to in a thread that is separate from the UI toolkit thread. It handles its own rendering and the timing of when that rendering happens.

One of the things that makes Renderscript compelling is that it can determine, at runtime, the best way to achieve high performance for a particular operation. For example, it might send graphics operations down to the GPU, or it might parcel out independent computations to separate cores in a multi-core device. All of this is transparent to the application; it simply starts the script, sets the values, and lets it go.

An important use case for Renderscript arises for SDK applications that need functionality or performance that cannot be had through the SDK directly. For example, your application may need to perform 3D graphics operations or heavy computation that takes too much time at the SDK level. You might consider taking these portions of your application and farming them out to Renderscript to do the work there instead.

An important consequence of the Renderscript approach using SDK and runtime-compiled code is that Renderscript applications are fully portable between devices. That is, you only need to compile your application once, not per-architecture (as you must do with the NDK). This single apk is then runnable on any device that supports the SDK level that you compiled for (3.0 and above, since 3.0 is the first Android release to support Renderscript).

Examples of Renderscript in Android 3.0 include the Live Wallpapers, the video wall view in the YouTube application, and the Books application (including that beautiful page-turn effect). Other examples can be found in the SDK samples; you should check these out to understand how to write Renderscript as well as how to initialize and interact with the scripts from SDK applications.

OpenGL Wrapper Functions

Android also provides wrappers for OpenGL APIs at the SDK level. That is, you can write an application using the SDK, with full access to usual SDK APIs and functionality, and still use OpenGL ES 1.x and OpenGL ES 2.0 APIs, by calling the wrapper functions in GLES10 or GLES20 classes. These wrappers call the underlying OpenGL APIs at the native level for those versions of OpenGL ES.

For casually exploring or using OpenGL, this may be a reasonable option. But while this approach works, it's not a recommended practice as a general approach for complex scenes that require high-performance graphics. For one thing, it is difficult to achieve high performance levels equivalent to native access to OpenGL due to the overhead of calling down from the SDK to the native level for every OpenGL call.

The Music application that shipped with Android 3.0 used this approach. The application was an SDK application which needed some simple 3D graphics operations, such as the carousel view of albums. The main reason that it uses the SDK/OpenGL approach is because it is an unbundled application (not dependent upon a particular release of the SDK) and needed to work on releases back to 2.2. It therefore used APIs that were available on those previous releases.

NDK

The NDK exists to provide an easy porting layer for existing applications written in native code, or which use native libraries. Porting might be more easily and quickly accomplished by using the NDK than by converting existing code to the language and APIs used by the SDK.

The NDK does not provide the rich GUI toolkit of the Android platform at the native level, so developers do not have access to the View objects, or the events, or the rest of the infrastructure that is provided in the SDK APIs. But there is a graphics environment at the NDK level that is sufficient for some specific purposes. In particular, game developers that simply want a fullscreen game experience can find what they need with OpenGL. This API provides low-level graphics functionality that lets applications display 2D and 3D graphics using the GPU for maximum graphics performance.

One important restriction of the NDK to keep in mind is that it compiles applications to specific CPU architectures. This means that if you only build your application for one specific chip, then the application will not work on Android devices that do not have that chip architecture. This is particularly important in the broad and growing Android ecosystem where new devices are coming out all the time. You probably want your application to work as well on new chips as it did on the ones you used to develop the application. So while the NDK OpenGL solution provides a reasonable route to fast graphics performance, it does so at the cost of the portability that other solutions offer.

One of the questions that came up about Renderscript when I talked to developers was how to access it from the NDK. This is not currently possible. Renderscript is specifically created to be a companion to SDK applications. We envision users of Renderscript as SDK applications that use Renderscript for their high-performance graphics or computation needs. These applications might be primarily SDK applications, with GUIs and interaction and most other facilities provided by the SDK APIs and small but important pieces of functionality provided by Renderscript, such as graphics effects that would otherwise not be possible or not perform as well. Or these applications might use Renderscript for most of what they do, just using the SDK to initialize the scripts and letting Renderscript take over from there.

And So...

I've tried to explain and categorize the different rendering options that we provide in the Android platform. What may seem confusing at first, because of the potential overlap, is actually simple once you understand how these libraries differ from each other in their purpose, their use, and their advantages. Here's one last brief take at a breakdown of when you might consider the various options:

  • SDK Canvas API: If you're writing a normal GUI application, using the SDK and the Canvas API is probably the most sensible option. If you need custom 2D graphics, there is plenty of power and reasonable performance in the Canvas API.
  • SDK Renderscript: If you want a fully portable, SDK application with access to the power, speed, and functionality of the Renderscript library and underlying GPU and multi-core CPU architectures, then it's worth exploring this new feature of Android.
  • SDK OpenGL Wrappers: If you are writing an SDK application and want to sprinkle in some 3D effects with OpenGL, you could see whether the OpenGL wrapper functions suit your needs.
  • NDK OpenGL: If you are porting your application from a different platform or existing native code and are looking for the fastest way to get it up and running on Android, then using the NDK with OpenGL is worth considering.

Whatever your rendering need, Android has something for you. So keep those cool graphics applications coming! (Sorry I still can't help you with the free tablet, though. There were tablets given away at Google I/O this year, so you could get one of those if you can travel back in time. I think there's an app for that. It probably uses Renderscript.)

Thursday, December 2, 2010

Slides from Recent Android Presentations

Here are slides for two of the approximately 871 talks that Romain Guy and I gave at Devoxx last month (and again at the SF Android User Group meeting a couple of days ago).

Videos of all of the Devoxx sessions are already posted on the excellent Parleys site for a small subscription fee, and should be made freely available sometime in the next few months. You probably want to check out those videos for the complete details. But in case you were at the talks, are sick of listening to us speak, or simply want to peruse the slideware, here you go...

Android Graphics and Animation

Android UI Development: Tips, Tricks and Techniques